Lesion Detection Artificial Intelligence Pipeline Computing System

ABSTRACT

A lesion detection and classification artificial intelligence (AI) pipeline comprising a plurality of trained machine learning (ML) computer models is provided. First ML model(s) process an input volume of medical images (VOI) to determine whether VOI depicts a predetermined amount of an anatomical structure. The AI pipeline determines whether criteria, such as a predetermined amount of an anatomical structure of interest being depicted in the input volume, are satisfied by output of the first ML model(s). If so, lesion processing operations are performed including: second ML model(s) processing the VOI to detect lesions which correspond to the anatomical structure of interest; third ML model(s) performing lesion segmentation and combining of lesion contours associated with a same lesion; and fourth ML models processing the listing of lesions to classify the lesions. The AI pipeline outputs the listing of lesions and the classifications for downstream computing system processing.

BACKGROUND

The present application relates generally to an improved data processingapparatus and method and more specifically to mechanisms for providingan artificial intelligence pipeline computing system that is able toidentify anatomical features of interest in electronic medical imagedata, such as lesions and the like.

Liver lesions are groups of abnormal cells in a biological entity'sliver, and may also be called a mass or tumor. Noncancerous, or benign,liver lesions are common and do not spread to other areas of the body.Such benign liver lesions do not generally cause any health issues.However, some liver lesions form as a result of cancer. Patients withcertain medical conditions may be more likely to have cancerous liverlesions than other patients. These medical conditions include hepatitisB or C, cirrhosis, iron storage disease (hemochromatosis), obesity orexposure to toxic chemicals such as arsenic or aflatoxin, for example.

Liver lesions are typically only identifiable by having a medicalimaging test, such as an ultrasound, magnetic resonance image (MRI),computerized tomograph (CT), or positron emission tomography (PET) scan,for example. Such medical imaging tests must be viewed by a humanmedical imaging subject matter expert (SME) who must use their ownknowledge and expertise as well as human ability to see patterns inimages, to determine if the medical imaging test shows any lesions. If apotentially cancerous lesion is identified by the human SME, thepatient's physician may have a biopsy performed to determine if thelesion is cancerous.

Abdominal contrast enhanced (CE) CT is the current standard inassessment of various abnormalities (e.g., lesions) in the liver. Theselesions may be evaluated by human SMEs as malignant (hepatocellularcarcinoma, cholangiocarcinoma, angiosarcoma, metastasis, and othermalignant lesions) or benign (hemangioma, focal nodular hyperplasia,adenoma, cyst or lipoma, granuloma, etc.). The manual evaluation of suchimages by human SMEs is important to guiding subsequent interventions.Many times, in order to properly evaluate lesions in a CE CT, amulti-phase study is conducted where the multi-phase study providesmedical imaging of different stages of enhancement of healthy liverparenchyma and comparison to enhancement of lesions to determinedifferences detection. The human SME can then determine a diagnosis ofthe lesions based on these differences.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described herein in the DetailedDescription. This Summary is not intended to identify key factors oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

In one illustrative embodiment, a method is provided, in a dataprocessing system, comprising at least one processor and at least onememory, the at least one memory comprising instructions executed by theat least one processor to implement a lesion detection andclassification artificial intelligence (AI) pipeline comprising aplurality of trained machine learning computer models. The AI pipelineexecutes the method comprising processing, by one or more first machinelearning computer models of the AI pipeline, an input volume of medicalimages to determine whether or not the input volume depicts apredetermined amount of an anatomical structure of interest. The methodfurther comprises determining, by logic of the AI pipeline, whether oneor more predetermined criteria are satisfied by output of the one ormore first machine learning computer models. The one or morepredetermined criteria comprises a predetermined amount of an anatomicalstructure of interest being depicted in the input volume.

In response to determining that the one or more predetermined criteriaare satisfied by the output of the one or more first machine learningcomputer models, the method executes lesion processing operations. Thelesion processing operations comprise processing, by one or more secondmachine learning computer models of the AI pipeline, the input volume todetect lesions present in medical images of the input volume whichcorrespond to the anatomical structure of interest. The lesionprocessing operations further comprise processing, by one or more thirdmachine learning computer models of the AI pipeline, the detectedlesions to perform lesion segmentation and combining of lesion contoursof different medical images in the input medical image associated with asame lesion, to generate a listing of lesions and their contours. Thelesion processing operations also comprise processing, by one or morefourth machine learning computer models of the AI pipeline, the listingof lesions and contours associated with the lesions in the listing oflesions, to classify the lesions in the listing of lesions into one ormore predetermined classes of lesions corresponding to the anatomicalstructure of interest.

The method also comprises outputting, by the AI pipeline, the listing oflesions and the classifications associated with the lesions fordownstream computing system processing. With this methodology improvedlesion detection and classification is made possible in images ofanatomical structures of interest. The methodology provides automatedcomputing tools that automatically identify lesions in anatomicalstructures and generates contours for these lesions while minimizingfalse positives and while correlating lesion contours associated withthe same lesion. Thus, a more accurate detection and classification oflesions is made possible over prior art mechanisms.

In some illustrative embodiments, the one or more first machine learningcomputer models of the AI pipeline comprises a body part detectionmachine learning model that executes on the input volume, detects aportion of a biological entity's body represented in the input volume,and determines whether the portion of the biological entity's bodycorresponds to a body part in which the anatomical structure of interestis present. In some illustrative embodiments, the body part is a humanabdomen, and the anatomical structure of interest is a human liver. Thisoperation allows the AI pipeline mechanisms to first ensure that thevolume of images being processed provide images of a portion of thebiological entity's body (e.g., abdomen) in which the particularanatomical structure of interest (e.g., the liver) would be present sothat processing resources are not applied to volumes of images where theanatomical structure is not present.

In some illustrative embodiments, the one or more first machine learningcomputer models of the AI pipeline comprises a phase classificationmachine learning model and an anatomical structure detection machinelearning model that operate on the input volume, wherein the phaseclassification machine learning model determines a phase of medicalimaging represented in each of the medical images of the input volume,and wherein the anatomical structure detection machine learning modeldetermines an amount of the anatomical structure of interest that isrepresented in the input volume. These operations ensure that the imagesof the volume all represent the same phase and thus, differences betweenimages due to different phases will not cause errors in lesiondetection. Moreover, determining whether there is a particular amount ofthe anatomical structure of interest present in the input volumeprovides another mechanism for ensuring that the images will provideuseful results by representing at least a minimum amount of theanatomical structure of interest so that a reliable result of whetherthe patient has lesions or not may be made by the AI pipeline.

In some illustrative embodiments, the one or more criteria furthercomprise the input volume having a single phase of medical imagingpresent in the input volume, and wherein based on results of anoperation of the phase classification machine learning model on theinput volume, and determining that a minimum threshold amount of theanatomical structure of interest is present in the input volume based onresults of an operation of the anatomical structure detection machinelearning model on the input volume. In some illustrative embodiments, inresponse to results of the operation of the phase classification machinelearning model indicating that more than one phase of medical imaging ispresent in the input volume, selecting a subset of medical images in theinput volume that correspond to a phase of interest and performing thelesion processing operation on only the selected subset of medicalimages in the input volume. Again, these operations ensure reliabilityof the results of the AI pipeline by eliminating potential sources oferror due to multiple phases being present in the input volume. Byproviding the functionality for selecting a subset of the volume thatare associated with a single phase, this allows adaptive processing ofthe input volume rather than having to reject the entire volume. In someillustrative embodiments, the phase of interest is one of a pre-contrastphase, an arterial contrast phase, portal/venous contrast phase, or adelayed phase.

In some illustrative embodiments, determining whether one or morepredetermined criteria are satisfied by output of the one or more firstmachine learning computer models further comprises executing, by axialscoring logic of the AI pipeline, an axial scoring operation on medicalimages in the input volume, wherein the axial scoring operationcomprises scoring images in the input volume in accordance with apredefined scoring algorithm to infer, from scores associated with theimages, axial scores for a most inferior slice in the volume (MISV) anda most superior slice in the volume (MSSV) and identifying a fraction ofthe anatomical structure of interest present in the input volume basedon the axial scores for the MISV and MSSV. These operations provide amechanism for determining the amount of an anatomical structure presentin the input volume. The scoring provides a measurement mechanism basedon the range of slices present in the input volume.

In some illustrative embodiments, the one or more second machinelearning computer models of the AI pipeline, comprises an ensemble ofmachine learning computer models, wherein each machine learning computermodel in the ensemble of machine learning computer models is trained bymachine learning processes to process the input volume in a differentmanner from other machine learning computer models of the ensemble ofmachine learning computer models, to generate corresponding lesiondetection predictions. By having an ensemble of differently trainedmachine learning computer models, the machine learning computer modelsmay compensate for the loss/error function focus and/or potentialweaknesses in training of other computer models of the ensemble andthereby improve the overall lesion detection.

In some illustrative embodiments, the ensemble of machine learningcomputer models comprises a mask generation machine learning model, aninput volume processing machine learning model, and a masked inputvolume processing machine learning model, wherein the mask generationmachine learning model is trained, by a machine learning process, togenerate a mask corresponding to the anatomical structure of interestand applies the mask to the input volume to generate a masked inputvolume, wherein the input volume processing machine learning model istrained, by a machine learning process, to generate a first lesionprediction output based on the input volume, and the masked input volumeprocessing machine learning model is trained, by a machine learningprocess, to generate a second lesion prediction output based on themasked input volume. By generating a mask and having a machine learningmodel that operates on a masked input volume, the processing performedby that machine learning model may focus on the portions of imagescorresponding to the anatomical structure of interest. Combining thismachine learning model with the operation of other machine learningmodels that operate on non-masked input allows for better lesiondetection.

In some illustrative embodiments, a first machine learning model of theensemble is trained with a first loss function that penalizes errors infalse negative classifications of lesions, and a second machine learningmodel of the ensemble is trained with a second loss function, differentfrom the first machine learning model, that penalizes errors in falsepositive classifications of lesions. By having multiple machine learningmodels implementing different loss functions, each machine learningmodel may compensate for the weaknesses of the other machine learningmodels with regard to sensitivity and specificity.

In some illustrative embodiments, logic of the ensemble applies a thirdloss function that compares a first lesion detection output of the firstmachine learning model to a second lesion detection output of the secondmachine learning model and operates to make the first lesion detectionoutput consistent with the second lesion detection output. The thirdloss function ensures that the outputs of the differently trainedmachine learning models of the ensemble are consistent with each otherduring training and thereby improves the training of the differentmachine learning models even though they are trained using differentloss functions

In some illustrative embodiments, the one or more third machine learningcomputer models of the AI pipeline, comprise a lesion segmentationmachine learning model that segments images of the input volume intocontours corresponding to lesions to generate lesion segments, a z-wiseconnection machine learning model that combines subsets of lesionsegments, based on the segments generated by the lesion segmentationmachine learning model, determined to be associated with a same lesionin three dimensional space, and a contour refinement machine learningmodel that isolates lesions in the input volume and refines contours ofthe lesions. These models allow for the identification of lesions inthree-dimensional space by connecting two-dimensional lesion contoursthat are associated with the same lesion and then refining the contoursof these lesions. Thus, a more accurate representation of detectedlesions is made possible in three-dimensions.

In some illustrative embodiments, one or more false positive removalmachine learning computer models are provided that operate on thelisting of lesions to remove false positive detections of lesions fromthe listing of lesions. The false positive detection helps to eliminatethose contours and regions of images in the input volume that aredetected as being lesions, but in fact are not lesions.

In some illustrative embodiments, the one or more false positive removalmachine learning computer models comprise one or more false positiveremoval machine learning computer models trained, by a machine learningprocess, based on two different operating points, wherein a firstoperating point of the two different operating points corresponds to apatient level operating point and a second operating point of the twodifferent operating points corresponds to a lesion level operatingpoint. The dual operating point implementation of false positive removalallows for more accurate false positive removal by first determining atthe patient level whether the patient has lesions, and if so, performinga more sensitive evaluation of the listing of lesions to remove lesionsin the listing of lesions that are not actual lesions.

In some illustrative embodiments, the one or more fourth learningcomputer models classify each of the lesions in the listing of lesionsinto one of a plurality of predetermined classes, wherein the one ormore predetermined classes comprises a type of lesion or at least one ofa benign class, malignant class, or indeterminate class. By classifyingthe lesions, medical personnel are informed of lesions that requiretreatment in the patient. Moreover, in some illustrative embodiments,this lesion classification may be used to process, by a downstreamcomputing system, the output of the listing of lesions and theclassifications associated with the lesions to generate a visualrepresentation of lesions in the listing of lesions in a graphical userinterface. In this way, medical personnel are provided with a visualrepresentation of the lesions that require their attention in order toprovide treatment to the patient.

In some illustrative embodiments, the input volume comprises imagescorresponding to at least one of a magnetic resonance imaging modalityor a computed tomography modality. Thus, the illustrative embodimentsmay be applied to known medical imaging technology as well as laterdeveloped medical imaging technology in which lesion detection andclassification is to be performed.

In other illustrative embodiments, a computer program product comprisinga computer useable or readable medium having a computer readable programis provided. The computer readable program, when executed on a computingdevice, causes the computing device to perform various ones of, andcombinations of, the operations outlined above with regard to the methodillustrative embodiment.

In yet another illustrative embodiment, a system/apparatus is provided.The system/apparatus may comprise one or more processors and a memorycoupled to the one or more processors. The memory may compriseinstructions which, when executed by the one or more processors, causethe one or more processors to perform various ones of, and combinationsof, the operations outlined above with regard to the method illustrativeembodiment.

These and other features and advantages of the present invention will bedescribed in, or will become apparent to those of ordinary skill in theart in view of, the following detailed description of the exampleembodiments of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention, as well as a preferred mode of use and further objectivesand advantages thereof, will best be understood by reference to thefollowing detailed description of illustrative embodiments when read inconjunction with the accompanying drawings, wherein:

FIG. 1 is an example block diagram of an AI pipeline implementingmultiple specifically configured and trained ML/DL computer models toperform anatomical structure identification and lesion detection ininput medical image data in accordance with one illustrative embodiment;

FIG. 2 is an example flowchart outlining an example operation of an AIpipeline in accordance with one illustrative embodiment;

FIG. 3A is an example diagram illustrating an example input volume ofslices (medical images) of an abdominal portion of a human patient inaccordance with one illustrative embodiment;

FIG. 3B shows another depiction of the input volume of FIG. 2A with asection of slices represented along with its corresponding axial scoress′_(inf) and s′_(sup);

FIG. 3C is an example diagram of the input volume of FIG. 2A in whichthe volume is divided axially into n fully overlapping sections;

FIGS. 4A-4C are example diagrams of one illustrative embodiment of theML/DL computer model configured and trained to estimate the s′_(sup) ands′i_(nf) values for a section of an input volume of medical images inaccordance with one illustrative embodiment;

FIG. 5 is a flowchart outlining an example operation of liver detectionand predetermined amount of anatomical structure determination logic ofan AI pipeline in accordance with one illustrative embodiment;

FIG. 6 is an example diagram of an ensemble of ML/DL computer modelsused to perform lesion detection in an anatomical structure of interest(e.g., the liver) in accordance with one illustrative embodiment;

FIG. 7 is a flowchart outlining an example operation of liver/lesiondetection logic in an AI pipeline in accordance with one illustrativeembodiment;

FIG. 8 depicts a block diagram of the aspects of lesion segmentation inaccordance with one illustrative embodiment;

FIG. 9 depicts a result of lesion detection and slice-wise partitioningin accordance with one illustrative embodiment;

FIGS. 10A-10D illustrate seed positioning in accordance with oneillustrative embodiment;

FIG. 11A is a block diagram illustrating a mechanism for lesionsplitting in accordance with one illustrative embodiment;

FIG. 11B is a block diagram illustrating a mechanism for seed relabelingin accordance with one illustrative embodiment;

FIG. 12 is a flowchart outlining an example operation of lesionsplitting in accordance with one illustrative embodiment;

FIGS. 13A-13C illustrate z-wise connection of lesions in accordance withone illustrative embodiment;

FIGS. 14A and 14B illustrate results of a trained model for z-wiselesion connection in accordance with one illustrative embodiment;

FIG. 15 is a flowchart outlining an example operation of a mechanism forconnecting two-dimensional lesions along the z-axis in accordance withone illustrative embodiment;

FIG. 16 illustrates an example with contours for two lesions in the sameimage in accordance with one illustrative embodiment;

FIG. 17 is a flowchart outlining an example operation of a mechanism forslice-wise contour refinement in accordance with one illustrativeembodiment;

FIG. 18A is an example of ROC curves for patient level and lesion leveloperating point determination in accordance with one illustrativeembodiment;

FIG. 18B is an example flow diagram of an operation for performing falsepositive removal based on patient and lesion level operating points inaccordance with one illustrative embodiment;

FIG. 18C is an example flow diagram of an operation for performingvoxel-wise false positive removal based on input volume level and voxellevel operating points in accordance with one illustrative embodiment;

FIG. 19 is a flowchart outlining an example operation of false positiveremoval logic of an AI pipeline in accordance with one illustrativeembodiment;

FIG. 20 is an example diagram of a distributed data processing system inwhich aspects of the illustrative embodiments may be implemented; and

FIG. 21 is an example block diagram of a computing device in whichaspects of the illustrative embodiments may be implemented.

DETAILED DESCRIPTION

The detection of lesions, or groups of abnormal cells, is largely amanual process in modern medicine. As this is a manual process, it isfraught with sources of error due to human limitations with regard toindividuals' abilities to detect portions of digital medical images thatshow such lesions, especially given the greater demands on suchindividuals to evaluate increasingly greater numbers of images inshorter amounts of time. While some automated image analysis mechanismshave been developed, there is still a need to improve such automatedimage analysis mechanisms to provide a more efficient and correctanalysis of medical image data to detect lesions in an imaged anatomicalstructure, e.g., the liver or other organs.

The illustrative embodiments are specifically directed to an improvedcomputing tool that provides automated computer driven artificialintelligence medical image analysis that is specifically trained,through machine learning/deep learning computer processes, to detectanatomical structures, detect lesions or other biological structures ofinterest in or associated with such anatomical structures, performspecialized segmentation of the detected lesions or other biologicalstructures, perform false positive removal based on the specializedsegmentation, and perform classification of the detected lesions orother biological structures, and provide the results of thelesion/biological structure detection to downstream computing systems toperform additional computer operations. The following description of theillustrative embodiments will assume embodiments specifically concernedwith the mechanisms of the illustrative embodiments being specificallytrained with regard to the liver lesions as the biological structure ofinterest, however the illustrative embodiments are not limited to such.Rather, those of ordinary skill in the art will recognize that themachine learning/deep learning based artificial intelligence mechanismsof the illustrative embodiments may be implemented with regard to aplethora of other types of biological structures/lesions in orassociated with other anatomical structures represented in medicalimaging data without departing from the spirit and scope of the presentinvention. Moreover, the illustrative embodiments may be described interms of the medical imaging data being computed tomography (CT) medicalimaging data, however the illustrative embodiments may be implementedwith any digital medical imaging data from various types of medicalimaging technologies including, but not limited to, positron emissiontomography (PET) and other nuclear medicine imaging, ultrasound,magnetic resonance imaging (MRI), elastography, photoacoustic imaging,echocardiography, magnetic particle imaging, functional near-infraredspectroscopy, elastography, various radiography imaging includingfluoroscopy, etc.

Overall, the illustrative embodiments provide an improved artificialintelligence (AI) computer pipeline comprising a plurality ofspecifically configured and trained AI computer tools, e.g., neuralnetworks, cognitive computing systems, or other AI mechanisms that aretrained based on a finite set of data to perform specific tasks. Theconfigured and trained AI computer tools are each specificallyconfigured/trained to perform a specific type of artificial intelligenceprocessing of a volume of input medical images, represented as one ormore collections of data and/or metadata that define the medical imagescaptured by medical imaging technology. In general, these AI toolsemploy machine learning (ML)/deep learning (DL) computer models (orsimply ML models) to perform tasks that, while emulating human thoughtprocesses with regard to the results generated, use different computerprocesses, specific to computer tools and specifically ML/DL computermodels, which learn patterns and relationships between data that arerepresentative of particular results, e.g., image classifications orlabels, data values, medical treatment recommendations, etc. The ML/DLcomputer model is essentially a function of elements including themachine learning algorithm(s), configuration settings of the machinelearning algorithm(s), features of input data identified by the ML/DLcomputer model, and the labels (or outputs) generated by the ML/DLcomputer model. By specifically tuning the function of these elementsthrough a machine learning process, a specific ML/DL computer modelinstance is generated. Different ML models may be specificallyconfigured and trained to perform different AI functions with regard tothe same or different input data.

As the artificial intelligence (AI) pipeline implements a plurality ofML/DL computer models, it should be appreciated that these ML/DLcomputer models are trained through ML/DL processes for specificpurposes. Thus, as an overview of the ML/DL computer model trainingprocesses, it should be appreciated that machine learning is concernedwith the design and the development of techniques that take as inputempirical data (such as medical image data), and recognizes complexpatterns in the input data. One common pattern among machine learningtechniques is the use of an underlying computer model M, whoseparameters are optimized for minimizing the cost function associated toM, given the input data. For instance, in the context of classification,the model M may be a straight line that separates the data into twoclasses (e.g., labels) such that M=a*x+b*y+c and the cost function wouldbe the number of misclassified points. The learning process thenoperates by adjusting the parameters a, b, c such that the number ofmisclassified points is minimal. After this optimization phase (orlearning phase), the model M can be used to classify new data points.Often, M is a statistical model, and the cost function is inverselyproportional to the likelihood of M, given the input data. This is justa simple example to provide a general explanation of machine learningtraining and other types of machine learning using different patterns,cost (or loss) functions, and optimizations may be used with themechanisms of the illustrative embodiments without departing from thespirit and scope of the present invention.

For purposes of anatomical structure detection and/or lesion detection(where lesions are “anomalies” in medical imaging data), a learningmachine may construct a ML/DL computer model of normal structurerepresentations, to detect data points in medical images that deviatefrom this normal structure representation ML/DL computer model. Forexample, a given ML/DL computer model (e.g., a supervised,un-supervised, or semi-supervised model) may be used to generate andreport anomaly scores to another device, generate classification outputsindicating one or more classes to which an input is classified,probabilities or scores associated with the various classes, or thelike. Example machine learning techniques that may be used to constructand analyze such ML/DL computer models may include, but are not limitedto, nearest neighbor (NN) techniques (e.g., k-NN models, replicator NNmodels, etc.), statistical techniques (e.g., Bayesian networks, etc.),clustering techniques (e.g., k-means, etc.), neural networks (e.g.,reservoir networks, artificial neural networks, etc.), support vectormachines (SVMs), or the like.

The processor-implemented artificial intelligence (AI) pipeline of theillustrative embodiments generally includes one or both of machinelearning (ML) and deep learning (DL) computer models. In some instances,one or the other of ML and DL can be used or implemented to achieve aparticular result. Traditional machine learning can include or usealgorithms such as Bayes Decision, Regression, Decision Trees/Forests,Support Vector Machines, or Neural Networks, among others. Deep learningcan be based on deep neural networks and can use multiple layers, suchas convolution layers. Such DL, such as using layered networks, can beefficient in their implementation and can provide enhanced accuracyrelative to traditional ML techniques. Traditional ML can bedistinguished from DL in general in that DL models can outperformclassical ML models, however, DL models can consume a relatively largeramount of processing and/or power resources. In the context of theillustrative embodiments, references herein to one or the other of MLand DL can be understood to encompass one or both forms of AIprocessing.

With regard to the illustrative embodiments, the ML/DL computer modelsof the AI pipeline are executed, after configuration and trainingthrough ML/DL training processes, and perform complex computer medicalimaging analysis to detect anatomical structures in input medical imagesand generate outputs specifically identifying target biologicalstructures of interest (hereafter assumed to be liver lesions forpurposes of description of example embodiments), their classifications,contours specifying where these target biological structures of interest(e.g., liver lesions) are present in the input medical images (hereafterassumed to be CT medical image data), and other information that aidshuman subject matter experts (SMEs), such as radiologists, physicians,and the like, in understanding a patient's medical condition from theviewpoint of the captured input medical images. Moreover, the outputscan be provided to other downstream computer systems to performadditional artificial intelligence operations, such as treatmentrecommendations and other decision support operations based on theclassifications, contours, and the like.

Initially, the artificial intelligence (AI) pipeline of the illustrativeembodiments receives an input volume of computed tomography (CT) medicalimaging data and detects which part of the biological entity's body isdepicted in the CT medical imaging data. A “volume” of medical images isa three-dimensional representation of the internal anatomical structureof the biological entity which is made up of stacks of two-dimensionalslices, where the slices may be individual medical images captured bymedical imaging technology. The stacks of slices may also be referred toas “slabs” and differ from the slices themselves in that the stacksrepresent a portion of the anatomical structure having a thickness, withthe stacking of slices or slabs generating a three-dimensionalrepresentation of the anatomical structure.

For purposes of this description, it will be assumed that the biologicalentity is a human being, however the present invention may operate onmedical images for various types of biological entities. For example, inveterinary medicine, the biological entity may be various types of small(e.g., pets such as dogs, cats, etc.) or large size animals (e.g.,horses, cows, or other farm animals). For implementations where the AIpipeline is specifically trained for the detection of liver lesions, theAI pipeline determines if the input CT medical imaging data representsan abdominal scan being present in the CT medical imaging data and ifnot, the operation of the AI pipeline terminates with regard to theinput CT medical imaging data as not being directed to the correct partor portion of the human body. It should be appreciated that there may bedifferent AI pipelines according to the illustrative embodiments,trained to process input medical images for different portions of thebody and different target biological structures, and the input CTmedical images may be input to each of the AI pipelines, or routed to anAI pipeline based on a classification of the body part or portion of thebody depicted in the input CT medical images, e.g., a classification ofthe input CT medical images as to body part or portion of the bodyrepresented in the input CT medical images may first be performed andthen a corresponding trained AI pipeline may be selected from aplurality of trained AI pipelines of the type described herein, toprocess the input CT medical images. For purposes of the followingdescription, a single AI pipeline trained to detect liver lesions willbe described, but the extension of this to a suite or collection of AIpipelines will be apparent to those of ordinary skill in the art in viewof the present description.

Assuming that the volume of input CT medical images comprises medicalimages of an abdominal portion of the human body (for purposes of liverlesion detection), further processing of the input CT medical images isperformed in two primary stages which may be performed substantiallyparallel to each other and/or in sequence depending on the desiredimplementation. The two primary stages comprise a phase classificationstage and an anatomical structure detection stage, e.g., liver detectionstage in the case where the AI pipeline is configured to perform liverlesion detection.

The phase classification stage determines if the volume of input CTmedical images comprises a single imaging phase or multiple imagingphases. The “phase” in medical imaging is an indication of a contrastagent uptake For example, in some medical imaging technologies, phasesmay be defined in terms of when a contrast medium is introduced into thebiological entity which allows for the capturing of medical images thatinclude capturing of the path of the contrast medium. For example,phases may include a pre-contrast phase, an arterial contrast phase, aportal/venous contrast phase and a delayed phase, with medical imagesbeing captured in any or all of these phases. Phases are usually relatedto timing after injection and to characteristics of the enhancement ofstructures within the images. Timing information can be taken intoaccount to “sort” potential phase (e.g. a delayed phase will always beacquired after a portal phase) and to estimate the potential phase(s) ofa given image. With regard to the use of characteristics of enhancementof structures within images, one example of using this type ofinformation to determine phase is described in commonly assigned andco-pending U.S. patent application Ser. No. 16/926,880, filed Jul. 13,2020, and entitled “Method of Determining Contrast Phase of aComputerized Tomography Image”. Additionally, timing information can beused in conjunction with other information (sampling, reconstructionkernel, etc.) to pick up the best representative of each phase (a givenacquisition can be reconstructed in several manners).

Once the images in the input volume are assigned or classified intotheir corresponding phases based on timing and/or characteristics ofenhancement, it can be determined based on the phase classificationwhether the volume comprises images of a single phase (e.g.,portal-venous phase present but no arterial) or multiphase exam (e.g., aportal-venous and arterial). If the phase classification indicates asingle phase is present in the volume of input CT medical images, thenfurther processing by the AI pipeline is performed as describedhereafter. If multiple phases are detected, then the volume is notfurther processed by the AI pipeline. However, in some illustrativeembodiments, while this filter of volumes based on single/multiplephases accepts only volumes with images from a single phase and rejectsmulti-phase volumes, in other illustrative embodiments the AI pipelineprocessing described herein may filter out images of a volume that arenot classified into a target phase of interest, e.g., the portal-venousphase images in the volume may be maintained while filtering out imagesof the volume that are not classified as being part of the portal-venousphase to thereby modify the input volume to be a modified volume havingonly a subset of images classified into the target phase. Moreover, asdiscussed previously, different AI pipelines may be trained fordifferent types of volumes and, in some illustrative embodiments, thephase classification of images within the input volume may be used toroute or distribute images of the input volume to corresponding AIpipelines that are trained and configured to process images of differentphases such that an input volume may be sub-divided into constituentsub-volumes and routed to their corresponding AI pipelines forprocessing, e.g., a first sub-volume corresponding to portal-venousphase images sent to a first AI pipeline while a second sub-volumecorresponding to arterial phase being sent to a second AI pipeline forprocessing. If the volume of input CT medical images comprises a singlephase, or after filtering and optionally routing of a sub-volume to acorresponding AI pipeline such that the AI pipeline(s) process images ofan input volume, or sub-volume, of a single phase, then the volume (orsub-volume) is passed onto a next stage of the AI pipeline for furtherprocessing.

The second primary stage is an anatomical structure of interest (whichin the example embodiments is the liver) detection stage in which theportions of the volume that depict the anatomical structure of interestare identified and passed to the next downstream stage of the AIpipeline. The anatomical structure of interest detection stage(hereafter referred to as the liver detection stage in accordance withthe example embodiment), comprises a machine learning (ML)/deep learning(DL) computer model that is specifically trained and configured toperform computerized medical image analysis to identify portions ofinput medical images that correspond to the anatomical structure ofinterest, e.g., a liver. Such medical image analysis may comprisetraining the ML/DL model on labeled training medical image data as inputto determine whether an input medical image (training image duringtraining) comprises the anatomical structure of interest, e.g., theliver. Based on a ground truth of the image labels, the operationalparameters of the ML/DL model may be adjusted to reduce a loss or errorin the results generated by the ML/DL model until convergence isachieved, i.e. the loss is minimized. Through this process, the ML/DLmodel is trained to recognize patterns of medical image data indicativeof the presence of the anatomical structure of interest (liver in theexample). Thereafter, once trained, the ML/DL model may be executed onnew input data to determine if the new input medical image data haspatterns indicative of the anatomical structure being present and if theprobability is greater than a predetermined threshold, it can bedetermined that the medical image data comprises the anatomicalstructure of interest.

Thus, in the liver detection stage, the AI pipeline uses the trainedML/DL computer model to determine if the volume of input CT medicalimages comprise images depicting the liver. The portions of the volumethat depict the liver are passed along with the results of the phaseclassification stage to a determination stage of the AI pipeline thatdetermines whether a single phase of medical imaging is present andwhether at least a predetermined amount of the anatomical structure ofinterest is present in the portions of the volume depicting theanatomical structure of interest (e.g., liver). The determination ofwhether a predetermined amount of the anatomical structure of interestis present may be determined based on known measurement mechanisms thatdetermine measurements of structures from medical images, e.g.,calculating sizes of structures from differences in pixel positionswithin an image. The measurements may be compared to predeterminedsizes, e.g., average sizes, of the anatomical structure for similarpatients having similar demographics such that if the measurementsrepresent at least a predetermined amount or portion of the anatomicalstructure then further processing may be performed by the AI pipeline.In one illustrative embodiment, this determination determines if atleast ⅓ of the liver is present in the portions of the volume of inputCT medical images that are determined to depict the liver, for example.While ⅓ is used in the example embodiments, any predetermined amount ofthe structure, determined to be appropriate for the particularimplementation, may be used without departing from the spirit and scopeof the present invention.

In one illustrative embodiment, in order to determine whether apredetermined amount of an anatomical structure of interest is presentin the volume of input CT medical images, an axial score is defined suchthat a slice corresponding to a medical image in the volume that has afirst representation of the anatomical structure of interest, e.g.,liver, i.e. the first slice containing the liver (FSL), is given a slicescore of 0 and the last slice containing the liver (LSL) has a scoreof 1. The first and last slice are defined, assuming a human biologicalentity, going from most inferior slice in volume (MISV) (closest to alower extremity, e.g., the feet) to the most superior slice in thevolume (MSSV) (closest to the head). The liver axial score estimate(LAE) is defined by a pair of slice scores, s_(sup) and s_(inf), whichcorrespond to slice scores for the MSSV and MISV slices, respectively. AML/DL computer model is specifically configured and trained to determinethe slice scores s_(sup) and s_(inf) for a volume of input CT medicalimages, as will be described in greater detail hereafter. Knowing theseslice scores and knowing from the definition above that the liverextends from 0 to 1, the mechanisms of the illustrative embodiments areable to determine the fraction of the liver in the field of view of thevolume of input CT medical images.

In some illustrative embodiments, the slice scores s_(sup) and s_(inf)may be found indirectly by first dividing the volume of input CT medicalimages into sections and then for each section executing the configuredand trained ML/DL computer model on the slices of the section toestimate a height for each slice in order to determine the most superior(closest to the head) and most inferior (closest to the feet) liverslice in the section s′_(sup) and s′_(inf). Given the estimates ofs′_(sup) and s′_(inf), estimates of s_(sup) and s_(inf) are found byextrapolation as it is known how the section is located with respect tothe entire volume of input CT medical images. This approach is based ona robust estimator of the height of an arbitrary slice from the inputvolume (or sub-volume associated with the target phase). Such anestimator can be obtained by learning a regression model, for instanceby using a deep learning model that performs the estimation of theheight from a chunk (set of consecutive slices). A long short-termmemory (LSTM) type artificial neural network model, for example, issuitable for these tasks are they have the capability to encode theordering of the slices containing the liver and abdomen anatomy. Itshould be noted that for each volume, there will be n number ofestimates of s_(sup) and s_(inf) where n is the number of sections pervolume. In one illustrative embodiment, the final estimate is obtainedby taking the unweighted mean of those n estimates, however in otherillustrative embodiments, the final estimate may be generated usingother functions of the n estimates.

Having determined the final estimates of the s_(sup) and s_(inf) for thevolume of input CT medical images, the fraction of the anatomicalstructure of interest, e.g., liver, is calculated based on these values.This task is made possible by the estimation of the height of eachslice. From an estimation of the height of the first (h1) and last (h2)slices of the liver in the input volume, assuming that the height of theactual first and last slices of the liver (whether they are contained inthe input volume or not) are H1 and H2, the portion of liver visible inthe input volume can be expressed as (min(h1, H1)−max(h2, H2))/(H1−H2).This calculated fraction may then be compared to a predeterminedthreshold value to determine if a predetermined minimum amount of theanatomical structure of interest is present in the volume of input CTmedical images or not, e.g., at least ⅓ of the liver is present in thevolume of input CT medical images.

If the determinations result in a determination that multiple phases arepresent and/or a predetermined amount of the anatomical structure ofinterest is not present in the portions of the volume of input CTmedical images depicting the anatomical structure, then furtherprocessing of the volume may be discontinued. If the determinationsresult in a determination that the volume of input CT medical imagescomprise a single phase and at least a predetermined amount of theanatomical structure of interest (e.g. ⅓ of the liver is shown in theimages), then the portions of the volume of input CT medical imageswhich depict the anatomical structure are forwarded to the next stage ofthe AI pipeline for processing.

In the next stage of the AI pipeline, the AI pipeline performs lesiondetection on the portions of the volume of input CT medical imagesrepresenting the anatomical structure of interest, e.g., the liver. Thisliver and lesion detection stage of the AI pipeline uses an ensemble ofML/DL computer models to detect the liver and lesions in the liver asrepresented in the volume of input CT medical images. The ensemble ofML/DL computer models uses differently trained ML/DL computer models toperform liver and lesion detection, with the ML/DL computer models beingtrained and using loss functions to counterbalance false positives andfalse negatives in lesion detection. Moreover, the ML/DL computer modelsof the ensemble are configured such that a third loss function forcesthe outputs of the ML/DL computer models to be consistent with eachother.

Assuming a liver detection and lesion detection being performed in thisstage of the AI pipeline, a first ML/DL computer model is executed onthe volume of input CT medical images to detect the presence of theliver. This ML/DL computer model may be the same ML/DL computer modelemployed in the prior AI pipeline stage of anatomical structure ofinterest detection, and thus, the results previously obtained may beutilized. A plurality (two or more) other ML/DL computer models areconfigured and trained to perform lesion detection in the portions ofthe medical images depicting the liver. A first ML/DL computer model isconfigured with two loss functions. The first loss function penalizeserrors in false negatives, i.e. classifications falsely indicating thatthere are no lesions present (normal anatomical structure). The secondloss function penalizes errors in false positive results, i.e.classifications falsely indicating that there are lesions present(abnormal anatomical structure). The second ML/DL is trained to detectlesions using an adaptive loss function which penalizes false positiveerrors in slices of the liver containing normal tissue, and penalizesfalse negative errors in slices of the liver containing lesions. Thedetection output from the two ML/DL models is averaged to produce afinal lesion detection.

The results of the liver/lesion detection stage of the AI pipelineincludes one or more contours (outlines) of the liver as well as adetection map identifying portions of medical imaging data elementscorresponding to detected lesions, e.g., a voxel-wise map of liverlesions detected in the volume of input CT medical images. The image mapis then input to a lesion segmentation stage of the AI pipeline. Thelesion segmentation stage, as will be described in greater detailhereafter, uses a watershed technique to partition the detection map togenerate image element, e.g., voxel, partitioning of the input CTmedical images. The liver lesion segmentation stage identifies all ofthe contours corresponding to lesions present in slices of the volume ofinput CT medical images based on this partitioning and performsoperations to identify which contours correspond to the same lesion inthree dimensions. The lesion segmentation aggregates correlated lesioncontours to generate three dimensional partitioning of lesions. Thelesion segmentation uses inpainting of lesion image elements, e.g.,voxels, and non-liver tissues represented in the medical images so as tofocus on each lesion individually and performs active contour analysis.In this way, individual lesions may be identified and processed withoutbiasing the analysis due to other lesions in the medical images orbiasing due to portions of the image outside the liver.

The result of the lesion segmentation is a listing of lesions with theircorresponding outlines or contours in the volume of input CT medicalimages. These output may contain findings that are not actual lesions.In order to minimize the impact of those false positives, the outputsare provided to a next stage of the AI pipeline directed to falsepositive removal using a trained false positive removal model. Thisfalse positive removal model of the AI pipeline acts as a classifier toidentify what outputs are actual lesions and what are the falsepositives from the detected findings. The input is composed of a volumeof images (VOI) around the detected finding associated with a maskresulting from the lesion segmentation refinement. The false positiveremoval model is trained using data that are the result of thedetection/segmentation stages: objects that are lesions from the groundtruth that are detected by the detection algorithm are used to representthe lesion class during training, while detections that do not match anylesion from the ground truth are used to represent the non-lesion (falsepositive) class.

In order to further improve the overall performance a dual operatingpoint strategy is employed on the lesion detection and false positivemodels. The idea is to note that the output of the AI pipeline can beinterpreted at different level. First, the output of the AI pipeline canbe used to tell whether an exam volume, i.e. input volume or volume ofimages (VOI), has a lesion or not. Second, the output of the AI pipelineaims at maximizing the detection of lesions regardless of whether theyare contained in a same patient/exam/volume or not. For the sake ofclarity, measurements made for an exam will be referred to herein as“patient level” and measures made for a lesion will be referred hereinas “lesion level.” Maximizing the sensitivity at “lesion level” willdegrade the specificity at “patient level” (one detection is enough fora patient to be said to contain lesions). This may end up beingsuboptimal for clinical usage as one would have to choose between havingpoor specificity at patient level, or low sensitivity at lesion level.

In view of this, the illustrative embodiments use a dual operating pointapproach for both lesion detection and false positive removal. Theprinciple is to run first the processing with a first operating pointthat gives reasonable performance at patient level. Then for patientsthat have at least on detected lesion from the first run, a secondoperating point is used to re-interpret/process the detected lesion(s).This second operating point is chosen to be more sensitive. While thissecond operating point is less specific than the first one, this loss ofspecificity is contained at patient level because all patients that hadno lesion detected with the first operating point are kept as is,regardless of whether the second operating point would have detectedadditional lesion(s) or not. Therefore, patient level specificity isdetermined only by the first operating point. The patient levelsensitivity is between one of the first and second operating pointstaken alone (one false negative case from the first operating point canbe turned into a true positive by the second operating point). On thelesion side, actual lesion level sensitivity is improved compared to thefirst operating point only. The lesion specificity is better than fromthe less specific second operating point taken alone as there are nofalse positives coming from cases processed only with the firstoperating point.

While the illustrative embodiments will assume particular configurationsand uses of the dual operating point approach, it should be appreciatedthat the dual operating point approach can be used with otherconfigurations and for other purposes where one is interested inmeasuring performance at group level (in the illustrative embodiments,this group level is “patient level”) and element level (in theillustrative embodiments, this element level is “lesion level”). While,in the illustrative embodiments, the dual operating point approach isapplied to both lesion detection and false positive removal, in can beappreciated that the dual operating point approach can be extendedbeyond these stages of the AI pipeline. For example, rather than patientlevel and lesion level, the detection of a lesion may be performed atvoxel level (element) versus volume level (group). As another example,the voxel or lesion level may be used for the element level and slabs(set of slices) may be used as the group level. In still a furtherexample, all the volume of an exam may be used as the group levelinstead of a single volume. It should be appreciated that this approachmay also be applied to two dimensional images (e.g., chest, mammography,etc. 2D xrays) for the images to be analyzed rather than threedimensional volumes. Specificity, such as the average number of falsepositives per patient/group, can be used for selecting the operatingpoint. Additionally, while the illustrative embodiments are described asapplying to lesion detection and classification, the dual operatingpoint based approach may be applied to other structures (clips, stents,implants, etc.) and beyond medical imaging.

The results of the dual operating point based detection and falsepositive removal lead to identification of the final filtered listing oflesions to be further processed by a lesion classification stage of theAI pipeline. In the lesion classification stage of the AI pipeline aconfigured and trained ML/DL computer model is executed on the listingof lesions and their corresponding contour data to thereby classify thelesions into one of a plurality of predetermined lesion classifications.For example, each lesion and its attributes, e.g., contour data, in thefinal filtered listing of lesions may be input to the trained ML/DLcomputer model which then operates on this data to classify the lesionas a particular type of lesion. The classification can be performedusing a classifier (e.g., trained neural network computer model)previously trained on ground truth data in combination with the resultof the previous processing steps of the AI pipeline. The classificationtask can be more or less complex, for instance it can be to provide alabel among benign, malignant or indeterminate, for example, or inanother example the actual lesion type, e.g., cyst, metastasis,hemangioma, etc. The classifier can be a neural network based computermodel classifier, e.g., SVM, decision tree, etc., or a deep learningcomputer model, for example. The actual input of this classifier is apatch around the lesion, which in some embodiments may be augmented witha lesion mask or outline (contour).

After classification of the lesions by the lesion classification stageof the AI pipeline, the AI pipeline outputs a list of lesions and theirclassifications, along with any contour attributes of the lesions.Moreover, the AI pipeline may also output the liver contour informationfor the liver. This AI pipeline generated information may be provided tofurther downstream computing systems for further processing andgeneration of representations of the anatomical structure of interestand any detected lesions present in the anatomical structure. Forexample, graphical representations of the volume of input CT medicalimages may be generated in a medical image viewer or other computerapplication with the anatomical structure and detected lesions beingsuperimposed or otherwise accentuated in the graphical representationusing the contour information generated by the AI pipeline. In otherillustrative embodiments, downstream processing of the AI pipelinegenerated information may include diagnosis decision support operations,automated medical imaging report generation based on the detectedlisting of lesions, classifications, and contour. In other illustrativeembodiments, based on classifications of lesions, different treatmentrecommendations may be generated for review and consideration by medicalpractitioners.

In some illustrative embodiments, the listing of lesions, theirclassifications, and contours may be stored in a history data structureassociated with a patient with which the volume of input CT medicalimages correspond such that multiple executions of the AI pipeline ondifferent volumes of input CT medical images associated with thatpatient may be stored and evaluated over time. For example, differencesbetween the listing of lesions and/or their associated classificationsand contours may be determined to evaluate the progression of a diseaseor medical condition of the patient and present such information to amedical professional for assisting with treatment of the patient.

Other downstream computing systems and processing of the specificanatomical structure and lesion detection information generated by theAI mechanisms of the illustrative embodiments may be implemented withoutdeparting from the spirit and scope of the present invention. Forexample, the output of the AI pipeline may be used by another downstreamcomputing system to process the anatomical structure and lesioninformation in the output of the AI pipeline to identify discrepancieswith other sources of information (e.g., a radiology report) in order tomake clinical staff aware of potential overlooked findings.

Thus, the illustrative embodiments provide mechanisms that provide anautomated AI pipeline comprising a plurality of configured and trainedML/DL computer models that implement various artificial intelligenceoperations for various stages of the AI pipeline so as to identifyanatomical structures and lesions associated with these anatomicalstructures in a volume of input medical images, determine the contoursassociated with such anatomical structures and lesions, determineclassifications of such lesions, and generate a listing of such lesionsas well as the contours of the lesions and the anatomical structures forfurther downstream computer processing of the AI generated informationfrom the AI pipeline. The operation of the AI pipeline is automated suchthat there is no human intervention at any of the stages of the AIpipeline and instead specially configured and trained ML/DL computermodels, trained through machine learning/deep learning computerprocesses, are employed to perform the specific AI analysis of thevarious stages. The only points at which human intervention may bepresent are prior to the input of the volume of input medical images,e.g., during medical imaging of the patient, and after output of the AIpipeline, e.g., viewing the augmented medical images presented via acomputer image viewing application based on the output of the list oflesions and contours generated by the AI pipeline. Thus, the AI pipelineperforms operations that cannot be performed by a human being as amental process and does not organize any human activity as the AIpipeline is specifically concerned with an improved automated computertool implemented as artificial intelligence using specific machinelearning/deep learning processes that only exist within a computerenvironment.

Before continuing the discussion of the various aspects of theillustrative embodiments and the improved computer operations performedby the illustrative embodiments, it should first be appreciated thatthroughout this description the term “mechanism” will be used to referto elements of the present invention that perform various operations,functions, and the like. A “mechanism,” as the term is used herein, maybe an implementation of the functions or aspects of the illustrativeembodiments in the form of an apparatus, a procedure, or a computerprogram product. In the case of a procedure, the procedure isimplemented by one or more devices, apparatus, computers, dataprocessing systems, or the like. In the case of a computer programproduct, the logic represented by computer code or instructions embodiedin or on the computer program product is executed by one or morehardware devices in order to implement the functionality or perform theoperations associated with the specific “mechanism.” Thus, themechanisms described herein may be implemented as specialized hardware,software executing on hardware to thereby configure the hardware toimplement the specialized functionality of the present invention whichthe hardware would not otherwise be able to perform, softwareinstructions stored on a medium such that the instructions are readilyexecutable by hardware to thereby specifically configure the hardware toperform the recited functionality and specific computer operationsdescribed herein, a procedure or method for executing the functions, ora combination of any of the above.

The present description and claims may make use of the terms “a”, “atleast one of”, and “one or more of” with regard to particular featuresand elements of the illustrative embodiments. It should be appreciatedthat these terms and phrases are intended to state that there is atleast one of the particular feature or element present in the particularillustrative embodiment, but that more than one can also be present.That is, these terms/phrases are not intended to limit the descriptionor claims to a single feature/element being present or require that aplurality of such features/elements be present. To the contrary, theseterms/phrases only require at least a single feature/element with thepossibility of a plurality of such features/elements being within thescope of the description and claims.

Moreover, it should be appreciated that the use of the term “engine,” ifused herein with regard to describing embodiments and features of theinvention, is not intended to be limiting of any particularimplementation for accomplishing and/or performing the actions, steps,processes, etc., attributable to and/or performed by the engine. Anengine may be, but is not limited to, software, hardware and/or firmwareor any combination thereof that performs the specified functionsincluding, but not limited to, any use of a general and/or specializedprocessor in combination with appropriate software loaded or stored in amachine readable memory and executed by the processor. Further, any nameassociated with a particular engine is, unless otherwise specified, forpurposes of convenience of reference and not intended to be limiting toa specific implementation. Additionally, any functionality attributed toan engine may be equally performed by multiple engines, incorporatedinto and/or combined with the functionality of another engine of thesame or different type, or distributed across one or more engines ofvarious configurations.

In addition, it should be appreciated that the following descriptionuses a plurality of various examples for various elements of theillustrative embodiments to further illustrate example implementationsof the illustrative embodiments and to aid in the understanding of themechanisms of the illustrative embodiments. These examples intended tobe non-limiting and are not exhaustive of the various possibilities forimplementing the mechanisms of the illustrative embodiments. It will beapparent to those of ordinary skill in the art in view of the presentdescription that there are many other alternative implementations forthese various elements that may be utilized in addition to, or inreplacement of, the examples provided herein without departing from thespirit and scope of the present invention.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Smalltalk, C++ or the like,and conventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a computer or other programmable data processing apparatusto produce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks. These computerreadable program instructions may also be stored in a computer readablestorage medium that can direct a computer, a programmable dataprocessing apparatus, and/or other devices to function in a particularmanner, such that the computer readable storage medium havinginstructions stored therein comprises an article of manufactureincluding instructions which implement aspects of the function/actspecified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Lesion Detection and Classification AI Pipeline Overview

FIG. 1 is an example block diagram of a lesion detection andclassification artificial intelligence (AI) pipeline (herein simplyreferred to as the “AI pipeline”) implementing multiple specificallyconfigured and trained ML/DL computer models to perform anatomicalstructure identification and lesion detection in input medical imagedata in accordance with one illustrative embodiment. For illustrativepurposes only, the depicted AI pipeline is specifically described asbeing directed to liver detection and liver lesion detection in medicalimage data. As noted above, the illustrative embodiments are not limitedto such and may be applied to any anatomical structure of interest andlesions associated with such anatomical structures of interest which maybe represented in image elements of medical image data captured bymedical imaging technology and the corresponding computing systems. Forexample, the mechanism of the illustrative embodiments may be applied toother anatomical structures, such as the lungs, heart, or the like, andthe detection, identification of contours, classification, and the like,of lesions associated with the lungs, heart, or other anatomicalstructure of interest.

Moreover, it should be appreciated that the following descriptionprovides an overview of the AI pipeline from the level illustrated inFIG. 1 and that subsequent sections of this description will go intoadditional detail regarding individual stages of the AI pipeline. Eachof the stages of the AI pipeline, in some illustrative embodiments, areimplemented as configured and trained ML/DL computer models, such as aneural network of deep learning neural network, as represented by thesymbol 103 in the various stages of the AI pipeline 100. These differentML/DL computer models are specifically configured and trained to performthe particular AI operations described herein, e.g., body partidentification, liver detection, phase classification, liver minimumamount detection, liver/lesion detection, lesion segmentation, falsepositive remove, lesion classification, etc. While these additionalsections of the following description will set forth specificembodiments for implementing the various stages of the AI pipeline whichprovide novel techniques, mechanisms, and approaches to performing theAI operations of the various stages, it should be appreciated that inthe context of the AI pipeline as a whole, other equivalent techniques,mechanisms, or approaches may be used without departing from the spiritand scope of the illustrative embodiments. These other equivalenttechniques, mechanisms, or approaches will be apparent to those ofordinary skill in the art in view of the present description and areintended to be within the spirit and scope of the present invention.

As shown in FIG. 1, the artificial intelligence (AI) pipeline 100, inaccordance with one illustrative embodiment, receives a volume of inputmedical images 105, which in the depicted example is a volume of inputcomputed tomography (CT) medical images represented as one or more datastructures, as input which is then automatically processed by thevarious stages of the AI pipeline 100 to ultimately generate an output170 that includes a listing of lesions along with their classificationsand contour information, and contour information about the anatomicalstructure of interest, e.g., the liver in the depicted example. Thevolume of input medical images 105 may be captured by medical imagingtechnology 102 using any of a plethora of generally known, or laterdeveloped, medical imaging techniques and equipment which render imagesof the internal anatomical structure of a biological entity, i.e. apatient, as one or more medical image data structures. In someillustrative embodiments, this volume of input medical images 105comprises two dimensional slices (individual medical images) of aportion of the patient's anatomical structure of a portion of theirbody, which are then combined to generate slabs (combinations of slicesalong an axis so as to provide a collection of medical images having athickness along the axis), and which are combined to generate a threedimensional representation, i.e. a volume, of the anatomical structuresof the portion of the body.

In a first stage logic 110 of the AI pipeline 100, the AI pipeline 100determines 112 the portion of the patient's body that corresponds to theinput volume of CT medical imaging data 105 and determines, via bodypart of interest determination logic 114, if this portion of thepatient's boy represents a portion of the patient's body thatcorresponds to the anatomical structure of interest, e.g., an abdominalscan rather than a cranial scan, lower body scan, or the like. Thisevaluation is to operate as an initial filter on the use of the AIpipeline 100 only with regard to volumes of input CT medical imagingdata 105 (hereafter referred to as the “input volume” 105) for which theAI pipeline 100 is specifically configured and trained to performanatomical structure identification and contouring and lesionidentification, contouring, and classification. This detection of thebody part represented in the input volume 105 may look to metadataassociated with the input volume 105 which may have fields specifyingthe region of the patient's body that was scanned, as may be specifiedby the source medical imaging technology computing system 102 whenperforming the medical imaging scans. Alternatively, the first stagelogic 110 of the AI pipeline 100 may implement a specifically configuredand trained ML/DL computer model for body part detection 112 thatperforms medical image classification with regard to particular portionsof patients' bodies that performs computerized pattern analysis on themedical image data of the input volume 105 and predicts a classificationof the medical imaging data with regard to one or more predeterminedportions of patient bodies. In some illustrative embodiments, thisevaluation may be binary, e.g., is or is not an abdominal medicalimaging volume, or may be a more complex multi-class evaluation, e.g.,specifically identifying probabilities or scores with regard to aplurality of different body part classifications, e.g., abdominal,cranial, lower extremities, etc.

If the body part of interest determination logic 114 of the first stagelogic 110 of the AI pipeline 100 determines that the input volume 105does not represent a portion of the patient's body where the anatomicalstructure of interest can be found, e.g., an abdominal portion of thebody where the liver can be found, then processing of the AI pipeline100 may be discontinued (reject case). If the body part of interestdetermination logic 114 of the first stage logic 110 of the AI pipeline100 determines that the input volume 105 does represent a portion of thepatient's body where the anatomical structure of interest can be found,then further processing of the input volume 105 by the AI pipeline 100is performed as described hereafter. It should be appreciated that insome illustrative embodiments, there may be multiple different instancesof the AI pipeline 100 provided, each configured and trained to processinput volumes 105 corresponding to different anatomical structures whichmay be present in different parts of the patient's body. Thus, the firststage logic 110 may be provided outside the AI pipeline(s) 100 and mayoperate as routing logic to route the input volume 105 to acorresponding AI pipeline 100 that is specifically configured andtrained to process input volumes 105 of particular classifications,e.g., one AI pipeline instance for liver and liver lesiondetection/classification, another AI pipeline instance for lung and lunglesion detection/classification, a third AI pipeline instance forcardiac and cardiac lesion detection/classification, etc. Thus, thefirst stage logic 110 may include routing logic that stores a mapping ofwhich AI pipeline instances 100 correspond to different bodyparts/anatomical structures of interest and, based on the detection ofthe body part represented in the input volume 105, may route the inputvolume 105 automatically to a corresponding AI pipeline instance 100that is specifically configured and trained to process input volumes 105corresponding to the detected body part.

Assuming that the input volume 105 is detected as representing a part ofthe patient's body where the anatomical structure of interest ispresent, e.g., an abdominal scan being present in the input volume 105for purposes of liver lesion detection, further processing of the inputvolume 105 is performed by the AI pipeline 100 in second stage logic120. This second stage logic 120 comprises two primary sub-stages 122and 124 which may be performed substantially parallel to each otherand/or in sequence depending on the desired implementation (parallelexecution being represented in FIG. 1 as an example). The two primarysub-stages 122, 124 comprise a phase classification sub-stage 122 and ananatomical structure detection sub-stage 124, e.g., liver detectionsub-stage 124 in the case where the AI pipeline 100 is configured toperform liver lesion detection.

The phase classification sub-stage 122 determines if the input volume105 comprises a single imaging phase, e.g., a pre-contrast phase, anarterial contrast phase, a portal/venous contrast phase, a delayedphase, etc. Again, the phase classification sub-stage 122 may beimplemented as logic that evaluates the metadata associated with theinput volume 105 that may include fields specifying the phases of themedical imaging study with which the medical images correspond as may begenerated by the medical imaging technology computing system 102 whenperforming the medical imaging. Alternatively, the illustrativeembodiments may implement a configured and trained ML/DL computer modelthat is specifically trained to detect patterns of medical images thatare indicative of different phases of a medical imaging study andthereby may classify the medical images of an input volume 105 as towhich phases they correspond. The output of the phase classificationsub-stage 122 may be binary indicating whether or not the input volume105 comprises one phase or multiple phases, or may be a classificationof each of the phases represented in the input volume 105 which can thenbe used to determine if a single phase or multiple phases arerepresented.

If the phase classification indicates a single phase is present in theinput volume 105, then further processing by the AI pipeline 100 throughthe downstream stages 130-170 is performed as described hereafter. Ifmultiple phases are detected, then the input volume 105 is not furtherprocessed by the AI pipeline 100, or as described previously, may befiltered and/or divided into sub-volumes, each sub-volume having imagesof a corresponding single phase such that only a sub-volumecorresponding to a target phase is processed by the AI pipeline 100and/or the sub-volumes are routed to corresponding AI pipelinesconfigured and trained to process input volumes of images correspondingto their particular phase classification. It should be appreciated thatan input volume can be rejected for several reasons (e.g., no liverpresent in the image, not a single-phase input volume, not enough liverpresent in the image, etc.). Depending on the actual root cause of therejection, the cause of the rejection may be communicated to a user viaa user interface or the like. For example, the output of the AI pipeline100 in response to the rejection may indicate the reason for therejection and may be utilized by a downstream computing system, e.g., aviewer or an additional automated processing system, to communicate thereason for the rejection through an output. For instance, in the case ofno liver being detected in the input volume, the input volume may besilently ignored, e.g., without communicating the rejection to the user,while for an input volume containing a liver, but comprising amultiphase input volume, the rejection may communicated to the user(e.g., radiologist) by clearly stating in a user interface generated bythe viewer downstream computing system that the input volume was notprocessed by the AI pipeline 100 due to the input volume having imagesof more than one phase, in order not to be mistaken with a input volumesthat do not contain any findings, for example.

The second primary sub-stage 124 is a detection sub-stage for detectingan anatomical structure of interest (which in the example embodiments isthe liver) in portions of the input volume 105. That is, the slices,slabs, etc. in the input volume 105, which specifically depict theanatomical structure of interest (liver), are identified and areevaluated to determine if a predetermined minimum amount of theanatomical structure of interest (liver) is present in these slices,slabs, or input volume as a whole. As mentioned previously, thedetection sub-stage 124 comprises a MU DL computer model 125 that isspecifically trained and configured to perform computerized medicalimage analysis to identify portions of input medical images thatcorrespond to the anatomical structure of interest, e.g., a human liver.

Thus, in the liver detection sub-stage 124, the AI pipeline 100 uses thetrained ML/DL computer model 125 to determine if the volume of input CTmedical images comprise images depicting the liver. The portions of thevolume that depict the liver are passed along with the results of thephase classification sub-stage 122 to a determination sub-stage 126,comprising single phase determination logic 127 and minimum structureamount determination logic 128, of the AI pipeline 100 that determineswhether a single phase of medical imaging is present 127 and whether atleast a predetermined amount of the anatomical structure of interest ispresent in the portions of the volume depicting the anatomical structureof interest (e.g., liver) 128. As mentioned previously, thedetermination of whether a predetermined amount of the anatomicalstructure of interest is present may be determined based on knownmeasurement mechanisms that determine measurements of structures frommedical images, e.g., calculating sizes of structures from differencesin pixel positions within an image, and compares these measurements toone or more predetermined thresholds to determine if a minimum amount ofthe anatomical structure of interest, e.g., liver, is present in theinput volume 105, e.g., ⅓ of the liver is present in the portions of theinput volume 105 that are determined to depict the liver, for example.

In one illustrative embodiment, in order to determine whether apredetermined amount of an anatomical structure of interest (liver) ispresent in the input volume 105, the previously described axial scoremechanism may be used to evaluate the portion of the anatomicalstructure present in the input volume 105. As described previously, aML/DL computer model may be configured and trained to estimate the slicescores, s_(sup) and s_(inf), which correspond to slice scores for theMSSV and MISV slices, respectively, for the input volume 105. In someillustrative embodiments, the slice scores s_(sup) and s_(inf) may befound indirectly by first dividing the input volume 105 into sectionsand then for each section executing the configured and trained ML/DLcomputer model on the slices of the section to estimate a slice scorefor the first and last slice in the section s′_(sup) and s′_(inf). Giventhe estimates of s′_(sup) and s′_(inf), estimates of s_(sup) and s_(inf)are found by extrapolation as it is known how the section is locatedwith respect to the entire volume of input CT medical images. It shouldbe noted that for each input volume 105, there will be n number ofestimates of s_(sup) and s_(inf) where n is the number of sections pervolume. In one illustrative embodiment, the final estimate is obtainedby taking the unweighted mean of those n estimates, however in otherillustrative embodiments, the final estimate may be generated usingother functions of the n estimates.

Having determined the final estimates of the s_(sup) and s_(inf) for thevolume of input CT medical images, the fraction of the anatomicalstructure of interest, e.g., liver, is calculated based on these values.This calculated fraction may then be compared to a predeterminedthreshold value to determine if a predetermined minimum amount of theanatomical structure of interest is present in the volume of input CTmedical images or not, e.g., at least ⅓ of the liver is present in thevolume of input CT medical images.

If the determinations of the determination logic 127 and 128 indicatethat multiple phases are present and/or a predetermined amount of theanatomical structure of interest is not present in the portions of theinput volume 105 depicting the liver, then further processing of theinput volume 105 by the AI pipeline 100 with regard to stages 130-170may be discontinued, i.e. the input volume 105 is rejected. If thedeterminations of the determination logic 127 and 128 result in adetermination that the input volume 105 has images of a single phase andat least a predetermined amount of the liver is depicted, then theportions of the input volume 105 which depict the anatomical structureare forwarded to the next stage 130 of the AI pipeline 100 forprocessing. While the example illustrative embodiment forwards the subportion of the input volume containing the liver for further processing,in other illustrative embodiments, context around the liver may also beprovided, which can be done by adding a predetermined amount of marginabove and below the selected liver area. Depending on how much contextis needed by the subsequent processing operations, that margin can beincreased up to the full coverage of the original input volume.

In the next stage 130 of the AI pipeline 100, the AI pipeline 100performs lesion detection on the portions of the input volume 105representing the anatomical structure of interest, e.g., the liver. Thisliver/lesion detection stage 130 of the AI pipeline 100 uses an ensembleof ML/DL computer models 132-136 to detect the liver and lesions in theliver as represented in the input volume 105. The ensemble of ML/DLcomputer models 132-136 uses differently trained ML/DL computer models132-136 to perform liver and lesion detection, with the ML/DL computermodels 132-136 being trained and using loss functions to counterbalancefalse positives and false negatives in lesion detection. Moreover, theML/DL computer models 132-136 of the ensemble are configured such that athird loss function forces the outputs of the ML/DL computer models132-136 to be consistent with each other.

In one illustrative embodiment, a configured and trained ML/DL computermodel 132 is executed on the input volume 105 to detect the presence ofthe liver. This ML/DL computer model 132 may be the same ML/DL computermodel 125 employed in the prior AI pipeline stage 120 and thus, theresults previously obtained may be utilized. A plurality (two or more)other ML/DL computer models 134-136 are configured and trained toperform lesion detection in the portions of the medical images of theinput volume 105 depicting the liver. A first ML/DL computer model 134is configured and trained to operate on the input volume 105 directlyand generate lesion predictions. A second ML/DL computer model 136 isconfigured with two different decoders that implement two different lossfunctions, one being a loss function that penalizes errors in falsenegatives, i.e. classifications falsely indicating that there are nolesions present (normal anatomical structure), and a second one being aloss function that penalizes errors in false positive results, i.e.classifications falsely indicating that there are lesions present(abnormal anatomical structure). The first decoder of the ML/DL computermodel 136 is trained to identify patterns representative of a relativelylarge number of different lesions at the cost of having a large numberof false positives. The second decoder of the ML/DL computer model 136is trained to be less sensitive to the detection of lesions, but thelesions that are detected are much more likely to be accuratelydetected. A third loss function of the ensemble of ML/DL computer modelsas a whole compares the results of the decoders of the ML/DL computermodel 136 to each other and forces them to be consistent with eachother. The lesion prediction results of the first and second ML/DLcomputer models 134, 136 are combined to generate a final lesionprediction for the ensemble, while the other ML/DL computer model 132that generates a prediction of a liver mask provides an outputrepresenting the liver and its contour. An example architecture of theseML/DL computer models 132-136 will be described in greater detailhereafter with regard to FIG. 6.

The results of the liver/lesion detection stage 130 of the AI pipeline100 includes one or more contours (outlines) of the liver, as well as adetection map identifying portions of medical imaging data elementscorresponding to detected lesions 135, e.g., a voxel-wise map of liverlesions detected in the input volume 105. The detection map is theninput to a lesion segmentation stage 140 of the AI pipeline 100. Thelesion segmentation stage 140, as will be described in greater detailhereafter, uses a watershed technique and corresponding ML/DL computermodel 142 to partition the detection map to generate image element,e.g., voxel, partitioning of the medical images (slices) of the inputvolume 105. The liver lesion segmentation stage 140 provides othermechanisms, such as ML/DL computer model 144, which identify all of thecontours corresponding to lesions present in slices of the input volume105 based on this partitioning, and performs operations to identifywhich contours correspond to the same lesion in three dimensions. Thelesion segmentation stage 140 further provides mechanisms, such as ML/DLcomputer model 146, which aggregate correlated lesion contours togenerate three dimensional partitioning of lesions. The lesionsegmentation uses in-painting of lesion image elements, e.g., voxels,and non-liver tissues represented in the medical images so as to focuson each lesion individually and performs active contour analysis. Inthis way, individual lesions may be identified and processed withoutbiasing the analysis due to other lesions in the medical images orbiasing due to portions of the image outside the liver.

The results of the lesion segmentation 140 is a listing of lesions 148with their corresponding outlines or contours in the input volume 105.These outputs 148 are provided to a false positive removal stage 150 ofthe AI pipeline 100. The false positive removal stage 150 uses aconfigured and trained ML/DL computer model that uses a dual operatingpoint strategy to reduce false positive lesion detections in the listingof lesions generated by the lesion segmentation stage 140 of the AIpipeline 100. A first operating point is selected to be sensitive tofalse positives by configuring the ML/DL computer model of the falsepositive removal stage 150 to remove as many lesions as possible. Adetermination is made as to whether a predetermined number or less ofthe lesions in the listing remain after the sensitive removal of falsepositives. If so, then the other lesions in the listing that wereremoved are reconsidered using a second operating point that isrelatively less sensitive to false positives. The results of bothapproaches identify the final filtered listing of lesions to be furtherprocessed by a lesion classification stage of the AI pipeline.

After having removed false positives from the list of lesions and theircontours generated by the lesion segmentation stage 140, the resultingfiltered list of lesions 155 is provided as input to a lesionclassification stage 160 of the AI pipeline 100 a configured and trainedML/DL computer model is executed on the listing of lesions and theircorresponding contour data to thereby classify the lesions into one of aplurality of predetermined lesion classifications. For example, eachlesion and its attributes, e.g., contour data, in the final filteredlisting of lesions may be input to the trained ML/DL computer model ofthe lesion classification stage 160 which then operates on this data toclassify the lesion as a particular predetermined type or class oflesion.

After classification of the lesions by the lesion classification stage160 of the AI pipeline 100, the AI pipeline 100 generates an output 170that includes a finalized list of lesions and their classifications,along with any contour attributes of the lesions. Moreover, the AIpipeline 100 output 170 may also include the liver contour informationfor the liver as obtained from the liver/lesion detection stage 130.This AI pipeline 100 generated output may be provided to furtherdownstream computing systems 180 for further processing and generationof representations of the anatomical structure of interest and anydetected lesions present in the anatomical structure. For example,graphical representations of the input volume may be generated in amedical image viewer or other computer application of the downstreamcomputing system 180 with the anatomical structure and detected lesionsbeing superimposed or otherwise accentuated in the graphicalrepresentation using the contour information generated by the AIpipeline. In other illustrative embodiments, downstream processing bydownstream computing systems 180 may include diagnosis decision supportoperations, automated medical imaging report generation based on thedetected listing of lesions, classifications, and contour. In otherillustrative embodiments, based on classifications of lesions, differenttreatment recommendations may be generated for review and considerationby medical practitioners. In some illustrative embodiments, the listingof lesions, their classifications, and contours may be stored in ahistory data structure of a downstream computing system 180 inassociation with a patient identifier such that multiple executions ofthe AI pipeline 100 on different input volumes 105 associated with thesame patient may be stored and evaluated over time. For example,differences between the listing of lesions and/or their associatedclassifications and contours may be determined to evaluate theprogression of a disease or medical condition of the patient and presentsuch information to a medical professional for assisting with treatmentof the patient. Other downstream computing systems 180 and processing ofthe specific anatomical structure and lesion detection informationgenerated by the AI pipeline 100 of the illustrative embodiments may beimplemented without departing from the spirit and scope of the presentinvention.

FIG. 2 is an example flowchart outlining an example operation of an AIpipeline in accordance with one illustrative embodiment. The operationoutlined in FIG. 2 may be implemented by the various stages of logic,including the configured and trained ML/DL computer models, shown inFIG. 1 and described above with specific example embodiments describedhereafter in the following separate sections of this description. Itshould be appreciated that this operation is specifically directed to anautomated artificial intelligence pipeline implemented in one or moredata processing systems having one or more computing devices which arespecifically configured to implement these automated computer toolmechanisms. Other than at medical image volume creation time, and use ofoutputs from downstream computing systems, there is no humanintervention in this outlined operation of FIGS. 1 and 2. The presentinvention is specifically providing improved automated artificialintelligence computing mechanisms to perform the described operationswhich avoid human interaction and reduce potential errors due toprevious manual processes by providing new and improved processes thatare specifically different than any previous manual process and arespecifically directed providing logic and data structures that permitthe improved artificial intelligence computing mechanisms of the presentinvention to be implemented in automated computing tools.

As shown in FIG. 2, the operation starts by receiving an input volume ofmedical images from a medical imaging technology computing system, e.g.,a computing system providing computer tomography (CT) medical images(step 210). The AI pipeline operates on the received input volume toperform body part detection (step 212) so that a determination may beperformed as to whether a body part of interest is present in thereceive input volume (step 214). If a body part of interest, e.g., theabdomen in the case of liver lesion detection and classification, is notpresent in the input volume, then the operation terminates. If the bodypart of interest is present in the input volume, phase classificationand minimum anatomical structure evaluations are performed, eithersequentially or in parallel.

That is, as shown in FIG. 2, phase classification is performed (step216) on the input volume to determine if the input volume comprisesmedical images (slices) for a single phase of medical imaging (e.g.,pre-contrast imaging, partial contrast imaging, delay phase, etc.) ormultiple phases. A determination is then made as to whether the phaseclassification indicates a single phase or multiple phases (step 218).If the input volume comprises medical images directed to multiplephases, the operation terminates; otherwise if the input volumecomprises medical images directed to a single phase, the operationcontinues to step 220.

In step 220, detection of the anatomical structure of interest, e.g.,the liver in the depicted examples, is performed so as to determinewhether a minimum amount of the anatomical structure is present in theinput volume to be able to perform accurately the subsequent stages ofthe AI pipeline operations. A determination is made as to whether aminimum amount of the anatomical structure is present or not, e.g., atleast ⅓ of the liver is represented in the input volume (step 222). Ifthe minimum amount is not present, the operation terminates; otherwise,the operation continues to step 224.

In step 224 liver/lesion detection is performed to generate contours anda detection map for the lesions. These contours and detection map areprovided to lesion segmentation logic which performs lesionsegmentation, e.g., liver lesion segmentation in the depicted example,based on these contours and detection map (step 226). The lesionsegmentation results in the generation of a listing of lesions and theircontours, as well as detection and contour information for theanatomical structure (e.g., liver) (step 228). Based on this listing oflesions and their contours, false positive removal operations areperformed on the lesions in the listing to remove false positives andgenerate a filtered listing of lesions and their contours (step 230).

The filtered listing of lesions and their contours are provided tolesion classification logic which performs lesion classification togenerate a finalized listing of lesions, their contours, and the lesionclassifications (step 232). This finalized listing is provided alongwith liver contour information to downstream computing systems (step234) which may operate on this information to generate medical imagingviews in a medical imaging viewer application, generate treatmentrecommendations based on classifications of detected lesions, evaluatehistorical progression of lesions over time for the same patient basedon a comparison of the finalized listing of lesions generated by the AIpipeline at different points in time, or the like.

Thus, the illustrative embodiments as outlined above provide automatedartificial intelligence mechanisms and ML/DL computer models thatoperate on an input volume of medical images and generates a listing oflesions, their contours, and classifications, while minimizing falsepositives. The illustrative embodiments provide automated artificialintelligence computer tools that specifically identify, in a given setof image voxels of the input volume, which ones of the voxels correspondto a portion of an anatomical structure of interest (e.g., the liver),and which ones of those voxels correspond to lesions in the anatomicalstructure of interest (e.g., liver lesions). The illustrativeembodiments provide a distinct improvement over previous approaches,both manual and automated, in that the illustrative embodiments can beintegrated in a fully automated computer tool in the clinician workflow.In fact, based on the early stages of the AI pipeline design of theillustrative embodiments, which to accept input volumes of only a singlephase, e.g., abdominal scans, and reject input volumes that do notdepict an anatomical structure of interest (e.g., liver), or do notdepict a predetermined amount of the anatomical structure of interest,(e.g., too small and amount of the liver), only the meaningful inputvolumes are processed through the automated AI pipeline, therebypreventing the radiologist spending valuable manual resources on uselessor faulty results when reviewing non-anatomical structure of interestinput volumes, e.g., non-liver cases. In addition to preventing floodingthe radiologist with useless information, the automated AI pipeline ofthe illustrative embodiments also ensure smooth information technologyintegration by avoiding congestion of the AI pipeline and downstreamcomputing systems such as network, archiving, and review computingsystems with data associated with cases that do not corresponding toanatomical structures of interest or cases that fail to provide asufficient amount of the anatomical structure of interest. Moreover, asdescribed above, the automated AI pipeline of the illustrativeembodiments allows for accurate detection, measurements andcharacterization of lesions in a fully automated manner, which is madetechnically possible by the automated AI pipeline structure and itscorresponding automated ML/DL computer model based components of one ormore of the illustrative embodiments.

ML/DL Computer Model for Detecting Minimum Amount of AnatomicalStructure Present in Input Volume

As noted previously, as part of the processing of the input volume 105,it is important to ensure that the input volume 105 represents a singlephase of medical imaging and that at least a minimum amount of theanatomical structure of interest is represented in the input volume 105.In order to determine if a minimum amount of the anatomical structure ofinterest is present in the input volume 105, in one illustrativeembodiment, the determination logic 128 implements a specificallyconfigured and trained ML/DL computer model that estimates slice scoresfor determining a portion of the anatomical structure, e.g., liver,present in the input volume 105. The following description provides anexample embodiment of this configured and trained ML/DL computer modelbased on a defined axial score technique.

FIG. 3A is an example diagram illustrating an example input volume(medical images) of an abdominal portion of a human patient inaccordance with one illustrative embodiment. In the depiction of FIG.3A, a two-dimensional representation of the three-dimensional volume isshown. Slices are horizontal lines within the two-dimensionalrepresentation shown in FIG. 3A, but would be represented as planesextending into and/or out of the page to represent a flattenedtwo-dimensional slice of the human body with the stacking of theseplanes resulting in a three-dimensional image

As shown in FIG. 3A, the illustrative embodiments define an axial scorefor slices that range from 0 to 1. The axial score is defined such thata slice corresponding to a first slice containing liver (FSL) has aslice score of 0 and a last slice containing liver (LSL) has a scoreof 1. In the depicted example, the first and last slices are defined inassociation with a most inferior slice in the volume (MISV) and a mostsuperior slice in the volume (MSSV), where inferior and superior aredetermined along a given axis of the volume, e.g., the y-axis in thedepicted example of FIG. 3A. Thus, in this depicted example, the MSSV isat a highest y-axis valued slice and the MISV is at a lowest y-axisvalued slice. For example, the MISV may be closest to a lower extremityof the biological entity, e.g., the feet of a human subject, and theMSSV may be closest to an upper portion of the biological entity, e.g.,the head of a human subject. The FLS is a slice depicting the anatomicalstructure of interest, e.g., the liver, that is relatively closest tothe MISV. The LSL is a slice depicting the anatomical structure ofinterest that is relatively closest to the MSSV. In one illustrativeembodiment, a trained ML/DL computer model, e.g., neural network, mayassign axial scores by taking chunks of slices as an input andoutputting the height (axial score) of the central slice in the chunk.This trained ML/DL computer model is trained with a cost function thatminimizes the error in the actual height (e.g., least square error).This trained ML/DL computer model is then applied to all the chunks thatcover the input volume (with possibly some overlap between the chunks).

The liver axial score estimate (LAE) is defined by a pair of slicescores, s_(sup) and s_(inf), which correspond to slice scores for theMSSV and MISV slices, respectively. A ML/DL computer model of thedetermination logic 128 in FIG. 1 is specifically configured and trainedto determine the slice scores s_(sup) and s_(inf) for an input volume105 and knowing these slice scores, the mechanisms of the illustrativeembodiments are able to determine the fraction of the liver in the fieldof view of the input volume 105.

In some illustrative embodiments, the slice scores s_(sup) and s_(inf)may be found indirectly by first dividing the input volume 105 intosections, e.g., sections comprising X number of slices, e.g., 20 slices,and then for each section executing the configured and trained ML/DLcomputer model on the slices of the section to estimate a slice scorefor the first and last slice in the section s′_(sup) and s′_(inf), where“first” and “last” may be determined in accordance with a progressiondirection along an axis of the three-dimensional volume 105, e.g., froma first slice to a last slice along a y-axis progressing from smallesty-axis value slice to highest y-axis value slice. Given the estimates ofs′_(sup) and s′_(inf), estimates of s_(sup) and s_(inf) are found byextrapolation as it is known how the section is located with respect tothe entire volume 105. It should be noted that for each volume, therewill be n number of estimates of s_(sup) and s_(inf) where n is thenumber of sections per volume. In one illustrative embodiment, the finalestimate is obtained by taking the unweighted mean of those n estimates,however in other illustrative embodiments, the final estimate may begenerated using other functions of the n estimates.

For example, FIG. 3B shows another depiction of the input volume of FIG.3A with a section of slices represented along with its correspondingaxial scores s′_(inf) and s′_(sup). As shown in FIG. 3B, a section isdefined as 20 slices 5 mm apart in this example. For each 20-slicesection of the volume, slice score s′_(sup) and s′_(inf) are estimatedby the ML/DL computer model and s_(sup) and s_(inf) are obtained from anextrapolation of these s′_(sup) and s′_(inf) values along a given range,e.g., a range from 0 to 1, a range from −0.5 to 1.2, or any otherdesired predetermined range suitable to the particular implementation.In this example, assuming a predetermined range of −0.5 to 1.2, ifs_(sup) is estimated by the application of the ML/DL computer model andthe extrapolation to be approximately 1.2 and s_(inf) is estimated to be−0.5 indicates that the entire liver is contained in the volume.Similarly, if the s_(sup) is estimated to be 1.2 and the s_(inf) is 0.5,these values indicate that approximately 50% of the superior axial liverextend is contained in the volume, e.g., coverage is(1.2−0.5))/(1.2−(−0.5))=0.41. As an additional example, in anotherillustrative embodiment in which the liver starts at −2.0 and ends at0.8 (i.e., s_(sup) is estimated to be 0.8 and s_(inf) is estimated to be2.0), the upper bound of the liver is lower than 1.2 so the liver is cuton the upper part and the lower bound is lower than −0.5 such that thebottom part of the liver is fully covered. This indicates thatapproximately 80% of the inferior axial liver extend is contained in thevolume, i.e. coverage is (0.8−max(−2, −0.5)/(1.2−(−0.5))=0.76.

FIG. 3C is an example diagram of the input volume of FIG. 3A in whichthe volume is divided axially into n fully overlapping sections. In thedepicted example, there are 7 sections indicated by arrows. It should benoted in this example that the last two sections (arrows at the top ofthe diagram) are almost completely the same. As mentioned previously,the s′_(sup) and s′_(inf) values are estimated by the ML/DL computermodel for each of these sections and used to extrapolate the s_(sup) andsi_(nf) values for the MSSV and MISV slices, which can then be used todetermine an amount of the anatomical structure of interest that ispresent in the input volume 105.

Thus, the s_(sup) and s_(inf) values for MSSV and MISV are foundindirectly by first dividing the input volume 105 into sections and thenfor each section estimating the slice scores for the first and lastslice in the section s′_(sup) and s′_(inf). Given these estimates, thevalues for s_(sup) and s_(inf) are estimated through an extrapolationsince it is known how the section is located with respect to the entireinput volume 105. There are n number of estimates of s_(sup) ands_(inf), extrapolated from each section, where n is the number ofsections per volume. The final estimate may be obtained, for example, byany suitable combinational function that evaluates the n estimates, suchas an unweighted mean of the n estimates or any other suitablecombinational function.

FIGS. 4A-4C show an example diagram of one illustrative embodiment ofthe ML/DL computer model configured and trained to estimate the s′_(sup)and s′i_(nf) values for a section of an input volume of medical imagesin accordance with one illustrative embodiment. The ML/DL computer modelof FIGS. 4A-4C is only one example of an architecture of the ML/DLcomputer model and many modifications to this architecture may be madewithout departing from the spirit and scope of the present invention,such as changing the tensor sizes of the input slices of the inputvolume, changing the number of nodes in the layers of the ML/DL computermodel, changing the number of layers, etc. Those of ordinary skill inthe art will recognize how to modify the ML/DL computer model of theillustrative embodiments to the desired implementation in view of thepresent description.

As shown in FIG. 4A, a sequence of 20 slices, representing a section 410or “slab” of the input volume 105 is provided as input to the processingblocks (PBs) 420-430. In the depicted illustrative embodiment, the PBs420-430 are blocks of logic that mix convolutional and LSTM layers, asshown in FIGS. 4B and 4C. Features are extracted from the convolutionallayers of the PB 420, 430, which are then fed as input to the LSTMlayers of the PB 420, 430. This is a type of smart/light modeling of thefact that slides have a particular order in the anatomical region oranatomical structure of interest, e.g., abdomen/liver, driven by theanatomy, e.g., relative position of the liver, kidney, heart, etc. inaddition to the liver anatomy itself. In the depicted example, initiallythe tensor size of the 20 slices 410 is 128×128 in this example. Thefirst processing block 420 reduces the size of the tensor by 8 in thisexample embodiment to generate a 20 slice section (it should beappreciated that the number of slices in a section is implementationspecific and can be modified without departing from the spirit and scopeof the present invention) with slices having dimensions 16×16×32, where32 is the number of filters. A second processing block 430 converts theinput section slices to a 20 slice section with slices having dimensions2×2×64, where 64 is the number of filters. A subsequent neural network440 configured with flattening, dense, and linear layers is configuredand trained to generate the s′_(sup) and s′_(inf) estimates for theinput section 410 of the input volume 105. FIG. 4B shows the compositionof the processing block (PB) with regard to the convolutional and LSTMlayers, and FIG. 4C shows example configurations of each of theseconvolutional and LSTM layers of each PB in accordance with oneillustrative embodiment.

With the ML/DL computer model architecture of FIGS. 4A-4C as an example,during training of this ML/DL computer model, in one illustrativeembodiment, medical imaging data, e.g., Digital Imaging andCommunications in Medicine (DICOM) data, is assembled into an inputvolume, e.g., a three dimensional array with S_(i)×512×512 size, float32, with Hounsfield Unit (HU) value, which is a normalized physicalvalue depicting the attenuation of the X-ray of the material beingpresented at a given location, e.g., voxel. S_(i) is the number ofslices in the i^(th) volume where i is in the range of 0 to N−1 with Nbeing the total number of volumes. Each input volume is processed by abody part detector and an approximate region corresponding to theabdomen is extracted (in the case of liver detection) as describedpreviously. The abdomen is defined as a continuous region between axialscores −30, 23, for example, from the body part detector. Slices outsidethis continuous region are rejected and the ground truth may be definedas locations of FSL and LSL that are appropriately adjusted. Forexample, assuming the input volume ranges from a to b, the input volumeis rejected if there is no overlap between [a:b] and [−30, 23]. In otherwords, if b>23 or if a<−30.

The input sections 410, or “slabs”, are resliced to a predeterminedslice separation, e.g., 5 mm. The input sections 410 are reshaped in x,y dimension to 128×128 which results in N sections 410 of shapeM_(i)×128×128. This is referred to as downsampling of the data in theinput volume. As the ordering of slices within an input volume relies onrough information, e.g., size of the organs, the AI pipeline stilloperates well on the downsampled data and both processing and trainingtime for the AI pipeline are improved due to the reduction in size ofthe downsampled data.

Input sections 410 which have less than a predetermined number ofslices, e.g., 20, or a pixel size smaller than a predetermined pixelsize, e.g., 0.55 mm, are rejected resulting in N′ of the M_(i)×128×128sections. Values in sections are clipped and normalized using lineartransformation from their acquisition range (e.g. −1024, 2048) to range(0, 1). At this point N′ of M_(i)×128×128 sections processed are, asdescribed above, constitute a training set upon which the neural network440 is trained to generate estimates of s′_(sup) and s′_(inf) for theinput sections.

With regard to performing inference with the trained neural network 440,the above operations for processing the input volume 105 through bodypart detection, slice selection corresponding to the body part ofinterest, re-slicing, reshaping, rejection of certain sections notmeeting predetermined requirements, and generating the clipped andnormalized sections are again performed with regard to new sections ofan input volume 105. After generating the clipped and normalizedsections, the input volume 105 is divided into R-ceil(M−10)/10sub-volumes, or sections, containing 20 slices, to thereby generate apartitioning of the slices with overlapping chunks. For example, ifthere is a N′=31 slices volume (slices numbered 0-30), three sections orsub-volumes are defined which contain the following overlapping slices:0-19, 10-29, 11-30. The sections or sub-volumes will typically have anoverlap of approximately at least 50%.

Thus, a ML/DL computer model is provided, configured, and trained so asto estimate the s_(sup) and s_(inf) values for an input volume based onestimates of the s′_(sup) and s′_(inf) values for sections of the volumecorresponding to a predetermined number of slices (medical images) givena defined axial score range from 0 to 1. From these estimates, adetermination can be made as to whether the input volume comprisesmedical slices that together constitute at least a predetermined amountof the anatomical structure of interest, e.g., the liver. Thisdetermination may be part of the determination logic 128 of the AIpipeline 100 for determining whether sufficient representation of theanatomical structure is present in the input volume 105 to allow foraccurate liver/lesion detection, lesion segmentation, etc. in furtherdownstream stages of the AI pipeline 100 as previously discussed above.

FIG. 5 is a flowchart outlining an example operation of liver detectionand predetermined amount of anatomical structure determination logic ofan AI pipeline in accordance with one illustrative embodiment. As shownin FIG. 5, the liver detection operation of the AI pipeline starts byreceiving the input volume (step 510) and divides the input volume intoa plurality of overlapping sections of a predetermined number of slicesfor each section (step 520). The slices for each section are input to atrained ML/DL computer model that estimates the axial scores for thefirst and last slices in each section (step 530). The axial scores forthe first and last slices are used to extrapolate to scores for the mostinferior slice in the volume (MISV) and the most superior slice in thevolume (MSSV) for the input volume (step 540). This results in aplurality of estimates for the axial scores for the MISV and MSSV whichare then combined through a function of the individual estimates tothereby generate an estimate of the axial scores for the MISV and MSSVof the input volume, e.g., a weighted mean of the like (step 550). Basedon the estimate of the axial score for the MISV and MSSV, the axialscores are compared to criteria for determining whether or not apredetermined amount of an anatomical structure of interest, e.g., theliver, is present in the input volume (step 560). Thereafter theoperation terminates.

Liver/Lesion Detection

Assuming that the input volume 105 is determined to have a single phaserepresented, and that the input volume 105 has a predetermined amount ofthe anatomical structure of interest represented in the slices of theinput volume 105, as described previously, liver/lesion detection isperformed on the portion of the input volume 105 comprising theanatomical structure of interest. The liver/lesion detection logic ofthe AI pipeline 100 stage 130, in one illustrative embodiment, employs aconfigured and trained ML/DL computer model that operates to detect theanatomical structure of interest (e.g., liver) in the slices of theinput volume 105 (again, in some illustrative embodiments, this may bethe same ML/DL computer model 125 used in stage 120 for liverdetection). The liver/lesion detection logic of the AI pipeline 100stage 130 also includes an ensemble of a plurality of other configuredand trained ML/DL computer models to detect lesions in images of theanatomical structure of interest (liver).

FIG. 6 is an example diagram of an ensemble of ML/DL computer modelsused to perform lesion detection in an anatomical structure of interest(e.g., the liver) in accordance with one illustrative embodiment. Theensemble of ML/DL computer models 600 comprises a first ML/DL computermodel 610 for detecting an anatomical structure of interest, e.g., theliver, and generating a corresponding mask. The ensemble of ML/DLcomputer models 600 also comprises a second ML/DL computer model 620that is configured and trained to process a liver masked input andgenerate lesion predictions using two competing loss functionsimplemented in two decoders of the second ML/DL computer model 620. Oneloss function is configured to penalize false positive errors (yieldinglow sensitivity, but high precision) and the other is configured topenalize false negative errors (yielding high sensitivity, but lowerprecision). An additional loss function, referred to as the consistencyloss 627 in FIG. 6, is employed for the second ML/DL computer model 620and forces the outputs generated by the two competing decoders to besimilar (consistent) to each other. The ensemble of ML/DL computermodels further includes a third ML/DL computer model 630 that isconfigured and trained to process the input volume 105 directly andgenerate lesion predictions.

As shown in FIG. 6 and noted above, the ensemble 600 comprises a firstconfigured and trained ML/DL computer model 610 that is specificallyconfigured and trained to identify an anatomical structure of interestin input medical images. In some illustrative embodiments, this firstML/DL computer model 610 comprises a U-Net neural network model 612configured and trained to perform image analysis to detect the liverwithin a medical image, however it should be appreciated that theillustrative embodiments are not limited to this particular neuralnetwork model and any ML/DL computer model that may perform segmentationmay be utilized without departing from the spirit and scope of thepresent invention. U-Net is a convolutional neural network developed forbiomedical image segmentation at the Computer Science Department of theUniversity of Freiburg, Germany. The U-Net neural network is based on afully convolutional network with an architecture modified and extendedto work with fewer training images and to yield more precisesegmentations. U-Net is generally known in the art and thus, a moredetailed explanation is not provided herein.

As shown in FIG. 6, in one illustrative embodiment, the first ML/DLcomputer model 610 may be trained to process a predetermined number ofslices at a time with the number being determined as appropriate for thedesired implementation, e.g., 3 slices was determined through empiricalprocesses to yield good results. The slices of the input volume, in oneillustrative embodiment were 512×512 pixel medical images, for example,although other implementations may use different slice dimensionswithout departing from the spirit and scope of the illustrativeembodiments. The U-Net generates segmentations of anatomical structuresin the input slices resulting in one or more segments corresponding toan anatomical structure of interest, e.g., the liver. The first ML/DLcomputer model 610 generates, as part of this segmentation, a segmentrepresenting a liver mask 614. This liver mask 614 is provided as aninput to at least one of the other ML/DL computer models 620 of theensemble 600 in order to focus processing by the ML/DL computer model620 on only the portion of input slices of the input volume 105corresponding to the liver. By pre-processing the input to the ML/DLcomputer model with a liver mask 614, the processing by the ML/DLcomputer model may be focused on the portions of the input slices thatcorrespond to the anatomical structure of interest and not on the“noise” in the input images. The others of the ML/DL computer models,e.g., ML/DL computer model 630, receives the input volume 105 directlywithout liver masking using the liver mask 614 generated by the firstML/DL computer model 610.

In the depicted ensemble 600 illustrative embodiment, the third ML/DLcomputer model 630 is composed of an encoder section 634-636 and adecoder section 638. ML/DL computer model 630 is configured to receive a9-slice slab of the input volume 105 which is then separated into groups631-633 of 3 slices each, with each group 631-633 being input to acorresponding encoder network 634-636. Each encoder 634-636 is aconvolutional neural network (CNN), such as a DenseNet-121 (D121),without the fully connected head, which has been pre-trained torecognize different types of objects (e.g., lesions) present in theinput slices and output classification outputs indicating the detectedtypes of objects present in the input slices, e.g., as an outputclassification vector or the like. The CNNs 634-636 may operate, forexample, on 3 channels of the input slices and the resulting outputfeatures of the CNNs 634-636 are provided to concatenation NHWC logic637, where NHWC refers to number of images in the batch (N), height ofthe image (H), width of the image (W), and number of channels of theimage (C). The architecture of the original DenseNet network comprisesmany convolutional layers and skip-true connections that down-sample the3-slice full resolution input to many feature channels with a smallerresolution. From then, a fully connected head aggregates all thefeatures and maps them to multiple classes in the final output of theDenseNet. Because the DenseNet network is used as an encoder in thedepicted architecture, the head is removed and only the down-sampledfeatures are kept. Then in the concatenation NHWC logic 637 all thefeature channels are concatenated to pass them into the decoder stage638 which has the role of up-sampling the images until a desired (e.g.512×512) output probability map resolution is reached.

The encoders 634-636 share the same parameters which are optimizedthrough the training process, e.g., the weights, sampling on lesiontypes during training, weights on the loss, type of augmentation, etc.The training of the ML/DL computer model 630 uses two different lossfunctions. The main loss function is an adaptive loss which isspecifically configured to penalize false positive errors in slices thatdo not have lesions in the ground-truth and also to penalize falsenegative errors in slices that have lesions in the ground-truth. Theloss function is a modified version of the Tversky loss as follows:

for each output slice:

TP=sum(prediction*target) FP=sum((1−target)*prediction)FN=sum((1−prediction)*target)

LOSS=1−((TP+1)/(TP+1+alpha*FN+beta*FP))

In which “prediction” is the output probabilities of the ML/DL computermodel 630 and “target” is the ground-truth lesion mask. Outputprobability values range between 0 and 1. Target has either 0 or 1 foreach pixel in the slice. For slices that do not have lesions in them,the “alpha” term is small (e.g., zero) and “beta” is large (e.g., 10).For slices that have lesions in them, “alpha” is large (e.g., 10) and“beta” is small (e.g., 1).

The second loss function 639 is one that is connected to the output ofthe encoders 634-636. Because the input for this loss is coming from themiddle of the ML/DL computer model 630, it is it is referred to as “deepsupervision” 639. Deep supervision has shown that it forces the encoderneural networks 634-636, during the training, to learn betterrepresentations of the input data. In one illustrative embodiment, thissecond loss is a simple mean square error to predict whether a slice hasa lesion in it or not. Hence a mapping network is used to map the outputfeatures of the encoders 634-636 to 9 values between 0 and 1 whichrepresent the probability of having a lesion in each of the 9-sliceinputs. The decoder 638 generates outputs specifying the probability mapfor detected lesions in the input images.

The second ML/DL computer model 620 receives a pre-processed input of 3slices from the input volume, which have been pre-processed with theliver mask 614 generated by the first ML/DL computer model 610 toidentify the portion of the 3 slices that corresponds to the liver mask614. The resulting pre-processed input slices (which are of size192×192×3 in the depicted example illustrative embodiment) are providedto the second ML/DL computer model 620 comprising a DenseNet-169 (D169)encoder 621 connected to two decoders (2D DEC—representing that thedecoders consist of 2-dimensional neural network layers). The D169encoder 621 is a neural network feature extractor, widely used incomputer vision applications. It consists of a series of convolutionallayers where features extracted from each layer are connected to anyother layer in a feed-forward fashion. The features extracted in theencoder 621 are transferred to two independent decoders 622, 623, whereeach decoder 622, 623 is composed of 2-dimensional convolution andup-sampling layers (referred to as 2D DEC in FIG. 6). Each decoder 622,623 is trained to detect lesions (e.g., liver lesions) in the inputslices. Although both decoders 622, 623 are trained to perform a sametask, i.e. lesion detection, a key difference in their training is thatthe two decoders 622, 623 each utilize a different loss function inorder to drive the detection training into two competing directions, asdiscussed previously and hereafter. The final detection map of thesecond ML/DL model 620 is combined with that of the third ML/DL model630 by means of the average operation 640. This procedure is appliedover all the input slabs of the input volume 105 to generate the finaldetection map (e.g. liver-lesions).

As mentioned above, the second ML/DL computer model 620 is trained usingtwo different loss functions, which attempt to achieve oppositedetection operating point performances. That is, where one of theencoders 622 uses a loss function for training that penalizes errors infalse negative lesion detection and thus, produces high sensitivitydetection with relatively low precision, the other of the encoders 623uses a loss function for training that penalizes errors in falsepositive lesion detection, resulting in low sensitivity detection butwith high precision. One example of these loss functions may be theFocal Tversky Loss (see Abraham et al., “A Novel Focal Tversky Lossfunction with Improved Attention U-Net for Lesion Segmentation,”arXiv:1810.07842[cs], October 2018) with parameters adjusted for high orlow penalty of false positives and false negatives in accordance withthe illustrative embodiments. A third loss function, the consistencyloss 627, is used to enforce consistency between the predicteddetections of each decoder 622, 623. The consistency loss logic 627compares the outputs 624, 625 of the two encoders 622, 623 to each otherand forces these outputs to be similar to each other. This loss may be,for example, a mean-squared error loss between the two predicteddetections, a structural similarity loss, or any other loss thatenforces consistency/similarity between the compared predicteddetections.

At run time, using these opposed operating point encoders 622, 623, thesecond ML/DL computer model 620 generates two lesion outputs 624, 625which are input to a slice averaging (SLC AVG) logic 623 that generatesan average of the lesion outputs. This average of the lesion outputs isthen re-sampled to generate an output commensurate in dimension to theoutput of the third ML/DL computer model 630 for comparison (please notethat this process consists of reverting the liver masking operation, andtherefore, computing the lesion output in the original 512×512×3resolution).

At run time, the slice averaging (SLC AVG) logic 626 operates on thelesion prediction outputs 624 and 625 of the encoders 622, 623 togenerate a final detection map of the ML/DL model 620. It should beappreciated that, while the consistency loss 627 was applied duringtraining to drive each decoder 622, 623 to learn consistent detections,at run time this consistency loss is no longer utilized and instead theML/DL model 620 outputs two detection maps that need to be aggregated bythe SLC AVG module 626. The results of the SLC AVG logic 626 areresampled to generate an output having dimensions commensurate with theinput slab (512×512×3). All the generated detections of the ML/DL model620 for each slab of the input volume 105 are combined with thegenerated detections of the ML/DL model 630 via the volume averaging(VOL AVG) logic 640. This logic computes the average of the twodetection masks at the voxel level. The result is a Final Lesion mask650 corresponding to the detected lesions in the input volume 105.

Thus, after training the ML/DL computer models 620, 630, when presentedwith new slices of a new input volume 105, the first ML/DL computermodel 610 generates the liver mask 614 for pre-processing the input tothe second ML/DL computer model 620, and the two ML/DL computer models620, 630 process the input slices to generate lesion predictions thatare averaged for the volume by the volume averaging logic 640. Theresult is a final lesion output 650 along with the liver mask output 660based on the operation of the first ML/DL computer model 610. Theseoutputs may be provided as the liver/lesion detection logic stage 130output of the AI pipeline 100 which is provided to the lesionsegmentation logic stage 140 of the AI pipeline 100 as previouslydiscussed above, and described in greater detail hereafter. Thus, themechanisms of the illustrative embodiments provide an ensemble 600approach to anatomical structure identification and lesion detection inan input volume 105 of medical images (slices).

With the ensemble architecture as shown in FIG. 6, improved performanceover the use of a single ML/DL computer model is achieved. That is, ithas been observed that through the use of the ensemble architecture, animproved detection specificity at a same level of sensitivity as asingle ML/DL computer model is achieved through the combining of thedetection outputs of the multiple ML/DL computer models of the ensemble.That is, with the ML/DL models 620, 630 making errors (false positives)in different locations, when their detection outputs are averaged, thesignal from the false positives decreases while the signal from the truepositive lesions prevails, leading to improved performance.

FIG. 7 is a flowchart outlining an example operation of liver/lesiondetection logic in an AI pipeline in accordance with one illustrativeembodiment. As shown in FIG. 7, the operation starts by receiving theinput volume (step 710) and performing anatomical structure detection,e.g., liver detection, using a first trained ML/DL computer model, suchas a U-Net computer model configured and trained to identify theanatomical structure (e.g., liver) (step 720). The result of theanatomical structure detection is a segmentation of the input volume toidentify a mask for the anatomical structure, e.g., liver mask (step730). The input volume is also processed via a first trained ML/DLcomputer model of an ensemble, which is specifically configured andtrained to perform lesion detection (step 740). The first trained ML/DLcomputer model generates a first set of lesion detection predictionoutputs based on its processing of the input volume (step 750).

A second trained ML/DL computer model of the ensemble receives a maskedinput generated by applying the generated anatomical structure mask tothe input volume and thereby identify portions of the medical images inthe input volume that correspond to the anatomical structure of interest(step 760). The second trained ML/DL computer model processes the maskedinput via two different decoders having two different and competing lossfunctions, e.g., one that penalizes errors in false positive lesiondetection and another that penalizes errors in false negative lesiondetection (step 770). The result is two sets of lesion predictionsoutputs which are then combined through combinational logic to generatea lesion prediction output of the second ML/DL computer model (step780). The second lesion prediction output is resampled if necessary andcombined with the first lesion prediction output generated by the firstML/DL computer model of the ensemble to generate a final lesionprediction output (step 790). The final lesion prediction output is thenoutput along with the anatomical structure mask (step 795) and theoperation terminates.

Lesion Segmentation

As described previously, through the operation of the various ML/DLcomputer models and stages of logic of the AI pipeline including bodypart detection, body part of interest determination, phaseclassification, anatomical structure of interest identification, andanatomical structure/lesion detection, a lesion prediction output isgenerated. For example, in the AI pipeline 100 shown in FIG. 1, theresults of the liver/lesion detection stage 130 of the AI pipeline 100includes one or more contours (outlines) of the liver, as well as adetection map identifying portions of medical imaging data elementscorresponding to detected lesions 135, e.g., a voxel-wise map of liverlesions detected in the input volume 105. The detection map is theninput to a lesion segmentation stage 140 of the AI pipeline 100.

As mentioned previously, the lesion segmentation logic, e.g., lesionsegmentation stage 140 in FIG. 1, uses a watershed technique andcorresponding ML/DL computer model to partition the detection map togenerate image element partitioning of the medical images (slices) ofthe input volume. The liver lesion segmentation stage also providesother mechanisms, such as one or more other ML/DL computer models, whichidentify all of the contours corresponding to lesions present in slicesof the input volume based on the image element partitioning, andperforms operations to identify which contours correspond to the samelesion in three dimensions. The lesion segmentation stage furtherprovides mechanisms, such as one or more additional ML/DL computermodels, which aggregate correlated lesion contours to generate threedimensional partitioning of lesions.

The lesion segmentation uses in-painting of lesion image elements andnon-liver tissues represented in the medical images so as to focus oneach lesion individually and performs active contour analysis. In thisway, individual lesions may be identified and processed without biasingthe analysis due to other lesions in the medical images or biasing dueto portions of the image outside the liver. The results of the lesionsegmentation is a listing of lesions with their corresponding outlinesor contours in the input volume.

FIG. 8 depicts a block diagram of an overview of the aspects of a lesionsegmentation process performed by the lesion segmentation logic inaccordance with one illustrative embodiment. As depicted in FIG. 8,lesion segmentation encompasses mechanisms for partitioningtwo-dimensional detections, i.e. detections of lesions intwo-dimensional slices, slice-wise (block 810), connectingtwo-dimensional lesions along the z-axis (block 820), and refiningcontours slice-wise (block 830). Each of these blocks will be describedin greater detail hereafter with regard to subsequent figures. Thesegmentation process shown in FIG. 8 is implemented as a process foridentifying all lesions in a given input volume under analysis anddistinguishing between lesions that are close to one another in theimages (slices) of the input volume. For example, two lesions thatappear to merge in pixel terms in one or more images may need to beidentified as two different regions or distinct lesions for purposes ofperforming other downstream processing of detected lesions, such asduring lesion classification, and in separate identification of lesionsin the output of a list of lesions for downstream computing systemoperations, such as providing a medical viewing application, performingtreatment recommendation operations, performing decision supportoperations, and the like.

As part of the partitioning of the 2D images slice-wise in block 810, amechanism of the illustrative embodiments uses an existing watershedtechnique to partition the detection map from the previous lesiondetection stage of the AI pipeline, e.g., the detection map 135generated by the liver/lesion detection logic 130 of the AI pipeline 100in FIG. 1. The watershed algorithm requires the definition of seeds toperform mask partitioning. It will split a mask into as many regions asthere are seeds such that every region has exactly one seed sittingroughly at its center as shown in FIGS. 10A and 10C. In automaticsegmentation, seeds in a mask can be obtained as the local maxima of itsdistance map (distance to the mask contour). However, such an approachis prone to noise and may lead to too many seeds, thereby over-splittingthe mask. Therefore, we need to edit the partition by regrouping some ofits regions. Considering the empirical observations that most lesionsare of a bubble shape, the guiding principle for region regrouping is tomake the resulting new region roughly circular. For example, for themask shown in FIG. 10C, the mechanism will merge the two regionsidentified respectively by seed 1051 and 1061, thereby producing a newmask partition comprising of only two roughly round regions. Thus for adetected lesion defined in detection map 135, such as the lesion shownon the left side of FIG. 9, described thereafter, it may be segmentedinto a few bubble-shaped lesions, as shown on the right side of FIG. 9.They will be interpreted as cross sections of 3D lesions on the slice.

Watershed segmentation is a region-based method that has its origins inmathematical morphology. In watershed segmentation, an image is regardedas a topographical landscape with ridges and valleys. The elevationvalues of the landscape are typically defined by the gray values of therespective pixels or their gradient magnitudes, thus considering atwo-dimensional as a three-dimensional representation. The watershedtransform decomposes an image into “catchment basins.” For each localminimum, a catchment basin comprises all points whose path of steepestdescent terminates at this minimum. Watersheds separate basins from oneanother. The watershed transform decomposes an image completely andassigns each pixel into either a region or a watershed.

Watershed segmentation requires selection of at least one marker,referred to as a “seed” point, interior to each object of the image.Seed points may be chosen by an operator. In one embodiment, the seedpoints are chosen by an automatic procedure that considers theapplication-specific knowledge of the objects. Once the objects aremarked, they can be grown using a morphological watershedtransformation, to be described in further detail below. Lesionstypically have a “bubble” shape. The illustrative embodiment provides atechnique for merging watershed partitioned regions based on thisassumption.

Thereafter, in block 820, a mechanism of the illustrative embodimentsaggregates the voxel partitioning on each of the slices along thez-direction to produce a three-dimensional output. Therefore, themechanism must determine if two sets of image elements, e.g., voxels, indifferent slices belong to the same lesion, i.e., whether they arealigned in three-dimensions. The mechanism computes measurements betweenlesions in adjacent slices based on intersection and union of thelesions and applies a regression model to determine whether the twolesions in the adjacent slices are part of the same region. One may vieweach lesion as a set of voxels, and the mechanism determines anintersect of two lesions as an intersect of the two sets of voxels anddetermines a union of two lesions as a union of the two sets of voxels.

This results in a three-dimensional partitioning of the lesions;however, the contours may not fit the actual image well. There may beover segmented lesions. The illustrative embodiments propose usingactive contouring, which is a traditional framework to tackle thesegmentation problem. Such an algorithm seeks to edit a contouriteratively to make it fit image data better and better and, in themeantime, ensure it maintains certain desirable properties such as shapesmoothness. In block 830, a mechanism of the illustrative embodimentsinitializes the active contours with partitioning obtained from thefirst and second stages 810, 820 and focuses on one lesion at a time;otherwise, running active contours or random segmentation methods onclose lesions may result in their being merged into one contour again,which is counterproductive because this amounts to essentially erasingthe benefits brought by the previous partitioning stage. The mechanismfocuses on one lesion and performs “inpainting” on lesion voxels andnon-liver tissues in the vicinity of the lesion under focus.

The chaining of these three stages of processing allows processing thatis unbiased by other lesions in the image or by pixels or lesionsoutside the liver.

Partitioning 2D Detections Slice-Wise

FIG. 9 depicts a result of lesion detection and slice-wise partitioningin accordance with one illustrative embodiment. As seen on the left sideof FIG. 9, a lesion area 910 is detected through the previous AIpipeline processes described above and may be defined in the output ofthe contours and detection map, e.g., 135 in FIG. 1, from the lesiondetection logic, e.g., 130 in FIG. 1. In accordance with oneillustrative embodiment, the logic of block 810 in FIG. 8 attempts topartition this region into three lesions 911, 912, and 913, as shown onthe right side of FIG. 9. The partitioning mechanism of the illustrativeembodiment is based on an existing watershed technique that operates topartition the detection map from the previous lesion detection stage ofthe AI pipeline. Watershed algorithms are used in image processingprimarily for segmentation purposes. The philosophy behind these knownwatershed algorithms is that grayscale images can be viewed as atopographic surface where high intensity denotes peaks and hills, whilelow intensity denotes valleys. The watershed technique starts fillingevery isolated valley (local minima) with different colored water(labels). As the water rises, depending on the peaks (gradients) nearby,water from different valleys, with different colors, will start tomerge. To avoid this, barriers are built in locations where watermerges. The work of filling water and building barriers continues untilall the peaks are under water at which point the barriers that werecreated gives the segmentation result. Again, watershed techniques aregenerally known and, thus, a more detailed description is not providedherein. Any known technique for partitioning 2D images slice-wise may beused without departing from the spirit and scope of the presentinvention.

In the context of lesion segmentation, the empirical observation thatmost lesions are of circular shape strongly suggests that a partitionthat results in a set of round regions is likely to be a good one.However, as previously stated, the quality of watershed type partitionhinges on that of the seeds. As a matter of fact, an arbitrary set ofseeds need not lead to a set of round regions. For instance, FIG. 10Cshows a watershed partition induced by 3 seeds that contains only oneroughly circular region. The other two are not circular. However, theirunion is again roughly circular. Such a configuration is termedover-splitting because the slanted split in the figure breaks anotherwise circular region into two smaller, no circular regions.Therefore, it is desirable to have an algorithm able to correctover-splitting. The seed relabeling mechanism does this by merging a fewover-split regions to form a coarser partition containing only roundshaped regions. For instance, the mechanism decides for the partition inFIG. 10C that merging two regions identified by seed 1051 and 1061 makesa more circular new region.

The illustrative embodiment merges regions in a partition into rounderand larger regions that may correspond to a physical lesion. Thepartition breaks an area into smaller regions, or as described herein,the partition breaks a mask into smaller regions. In terms of contours,a partition thus produces a set of smaller contours from a large contour(see FIG. 9 left to right).

Seeds are obtained by extracting local maxima from a distance mapcomputed from an input mask to partition. The map measures for eachpixel its Euclidean distance to the mask contour. Depending on thetopology of the input mask, local maxima derived from this distance mapmay lead to an overly fragmented partition by watershed algorithms. Inthis case, watershed is said to over-split and tends to produce regionsthat are not circular, which may be desirable in some applications, butis not ideal for lesion segmentation. In FIG. 10C, we show a syntheticinput mask whose distance map has three local maxima. Watershed thusresults in a partition which contains three regions, of which only oneis roughly circular (corresponding to seed 1071). The other two are not.The region having seed 1051 is only half circular. The seed relabelingmechanism then checks all seed pairs and determines that two regionscorresponding to seed 1051 and 1061 should be merged as together theywill form a more perfect bubble. Such an operation leads to a newpartition containing only two regions, and both are roughly circular inshape.

Local maxima are points that have the largest distance to the contourcompared to its immediate neighbor. A local maximum is a point, and itsdistance to the contour is known. As a result, the mechanism of theillustrative embodiment can draw a circle centered on this point. Thecircle's radius is the distance. For two local maxima, the mechanism canthus compute the overlap their respective circles. This is depicted inFIG. 10A and FIG. 10B.

The seed relabeling determines whether to merge two regions as follows.For two regions whose associated seeds are immediate neighbors, themerge will occur; otherwise, the mechanism bases its decision on ahypothesis testing procedure. For example, with reference to FIG. 10A,the depicted example describes a situation where a distance map yieldstwo distinct local maxima, which leads to the hypothesis that eachmaximum represents the center of a distinct circular lesion. Note thatthe distance map also allows the mechanisms of the illustrativeembodiments to tell how far the maxima are from the contour (boundary).This distance is represented in FIG. 10B by the dotted segmentsconnecting a maximum and a point on the contour. Therefore, if thehypothesis holds, one can infer the spatial extents of these two lesionsdue to the assumption that lesions are of approximately round or“bubble” shape. This allows the mechanisms of the illustrativeembodiments to draw two completed circles as shown in FIG. 10B. Fromthis, the mechanism then measures the two circles' overlap (e.g., withthe classic dice metric) and compares it to a pre-determined threshold.If the overlap metric's value is greater than the threshold, themechanism concludes that the two bubbles overlap too much to be distinctand a merge will take place. In other words, the mechanism of theillustrative embodiment then concludes that the two local maximacorrespond to two “centers” of the same lesion. However, in conventionalwatershed, no such seed (i.e., maxima) relabeling mechanism exists.Thus, mask over-splitting occurs often.

The overlap can be measured in a number of ways. In one exampleembodiment, the mechanism uses the dice coefficient. For two fullcircles corresponding to two local maxima as shown in FIG. 10B, themechanism may calculate the dice metric of these two circles. In thisway, the mechanism may learn from the training dataset what the optimalthreshold to apply in practice such that once the dice metric is greaterthan the threshold, the two local maxima are actually centers of thesame lesion.

FIGS. 10C and 10D provide an example of another lesion mask shape whichdiffers from that of FIGS. 10A and 10B in that two partially mergingcircles are closer to one another in FIG. 10A than in FIG. 10C. Due tothe distance map, which may be very sensitive to the mask shape, thereare three seeds in the example lesion mask shape of FIG. 10C. Followingthe above reasoning, the lesion splitting algorithm will split thelesions represented in FIG. 10C into two separate lesions, but not threeseparate lesions as would occur in the watershed technique without seedrelabeling.

In FIGS. 10C and 10D, seeds 1051 and 1061 represent a more extreme casethan the one described in FIGS. 10A and 10B. Without seed relabelingtechnique of the illustrative embodiments, a splitting will take place(represented by the slanted solid line) to separate them. But with theseed relabeling mechanism of the illustrative embodiments, thisundesirable outcome can be effectively avoided. To the contrary, sinceseed 1071 lies sufficiently far away from seed 1051 and 1061, the samehypothesis testing procedure described above will help accept thehypothesis that seed 1071 corresponds to the center of a distinctbubble, leading to the vertical split as shown in FIGS. 10C and 10D.Equivalently, this translates into a different label for seed 1071 fromthe one assigned to seed 1051 and 1061. However, similar to thesituation in FIGS. 10A and 10B, the hypothesis testing procedure of theseed relabeling technique of the illustrative embodiments will determinethat the seeds 1051 and 1061 correspond to the same lesion.

FIG. 11A is a block diagram illustrating a mechanism for lesionsplitting and relabeling in accordance with one illustrative embodiment.As shown in FIG. 11A, the mechanism, which may be implemented as acomputer model comprising one or more algorithms, machine learningcomputer models, and the like, executed by one or more processors of oneor more computing devices and which operates on input volumes of one ormore medical image data structures, receives a two-dimensional lesionmask 1101 and performs a distance transform (block 1102) to generatedistance map 1111. The distance transform (block 1102) is an operationperformed on a binary mask which computes, for each point in the lesionmask, its shortest distance to the mask contour (boundary). The more onemoves towards the interior of the lesion mask, the further one will beaway from its contour (boundary). Thus, the distance transformidentifies the lesion mask's center points, i.e., those points with alarger distance than others. In one embodiment, the mechanism optionallyperforms Gaussian smoothing on the distance map 1111.

The mechanism then performs local maxima identification (block 1103) togenerate seeds 1112. As described above, these local maxima are thepoints in the distance map 1111 that have a highest distance from acontour or boundary. The mechanism performs the watershed technique(block 1104) based on seeds 1112 to generate a watershed split lesionmask 1113. As explained above, this split lesion mask 1113 may beover-split, resulting in regions that do not conform to the assumedbubble shape of a lesion. Therefore, the mechanism performs seedrelabeling (block 1120) based on the distance map 1111, seeds 1112, andthe split 2D lesion mask 1113 to generate updated split lesion mask1121. The seed relabeling is described in further detail below withreference to FIG. 11B. The resulting updated split lesion mask 1121 willhave regions that have been merged to form regions that more accuratelyconform to bubble shapes assumed for lesions.

FIG. 11B is a block diagram illustrating a mechanism for seed relabelingin accordance with one illustrative embodiment. As shown in FIG. 11B,the mechanism, which may be implemented as a computer model comprisingone or more algorithms, machine learning computer models, and the like,executed by one or more processors of one or more computing devices andwhich operates on input volumes of one or more medical image datastructures, receives distance map 1111 and seeds 1112. Morespecifically, the mechanism considers each pair of seeds (seed A andseed B) in seeds 1112. The mechanism determines whether seed A and seedB are immediate neighbors (block 1151). If seed A and seed B areimmediate neighbors, then the mechanism assigns seed A and seed B thesame label (block 1155). In other words, seed A and seed B are groupedto represent a single region.

If seed A and seed B are not immediate neighbors in block 1151, then themechanism performs spatial extent estimation (1152) based on thedistance map 1111 and determines pairwise affinity for seed A and seedB, as described below. In accordance with the illustrative embodiment,spatial extent estimation assumes that a region is a “bubble” shape.Thus, the mechanism assumes each seed represents a circle with thedistance from the distance map as the radius of the circle.

Then, the mechanism computes an overlap metric (block 1153) for thecircles represented by seed A and seed B. In one example embodiment, themechanism uses a dice metric as follows:

$\frac{2{❘{A\bigcap B}❘}}{\left\lceil A \right\rceil + {❘B❘}}$

where |A| denotes the area of the circle represented by seed A, |B|denotes the area of the circle represented by seed B. Similarly, |A∩B|denotes the area of the intersection of A and B. In an alternativeembodiment, the mechanism may compute the overlap metric as follows:

$\frac{❘{A\bigcap B}❘}{❘{A\bigcup B}❘}$

where |A| denotes the area of the circle represented by seed A, |B|denotes the area of the circle represented by seed B, |A∩B| denotes thearea of the intersection of A and B, and |A∪B| denotes the area of theunion of A and B.

The mechanism determines whether the overlap metric is greater than apredetermined threshold (block 1154). If the overlap metric is greaterthan the threshold in block 1154, then the mechanism merges thecorresponding regions (block 1155) in split 2D lesion mask 1113.

If the affinity between two seeds is greater than the threshold, thenthey are assigned the same label. Otherwise, at this stage, it is notknown whether they should belong to the same group or not. This decisionis left to the label propagation stage (block 1512 in FIG. 15), the samemodule used in z-wise connection, to be described below.

In a situation where we have more than two seeds, the same operation ofFIG. 11B is repeated for all seed pairs before label propagation, whichproduces seed groups. For instance, there are situations where seed pair(a, b) and (b, c) are determined to belong to the same group, whereasseed pair (a, c) fails the test as shown in FIG. 11B. Then the labelpropagation will have to put a, b, c in the same group, that is, regionscorresponding to seeds a and c will still merge. However, if there areseeds a, b, c, and d and affinity computation (performed on six pairs intotal) shows that only (a, b) and (c, d) pass the test, then labelpropagation would result in two groups, containing respectively (a, b)and (c, d). Therefore, if a seed pair fails the test, it means that itis not known whether they should be put in the same group, but not theyshould belong to different groups.

For example, in FIG. 10C, there are 3 seed pairs (1051-1061, 1051-1071,1061-1071) and the mechanism should determine that seed 1051 and 1061should be assigned the same label (belonging to the same group). A labelpropagation step will then cluster these 3 seeds into 2 groups, thefirst containing only 1071 and the second having both 1051 and 1061.

FIG. 12 is a flowchart outlining an example operation for lesionsplitting in accordance with an illustrative embodiment. The operationoutlined in FIG. 12 may be performed by the mechanisms described abovewith regard to FIGS. 11A-11B. As shown in FIG. 12, the operation begins(step 1200), and the mechanism generating a distance map for atwo-dimensional lesion mask (block 1101). As discussed above, thisdistance map may be generated by performing a distance transformationoperation on the two-dimensional lesion mask and optionally performingGaussian smoothing to remove noise. The mechanism then uses local maximaidentification to generate groupings of data points, e.g., the localmaxima for each group (step 1202). The mechanism executes lesionsplitting based on local maxima to generate regions (step 1203). Themechanism then uses the distance map to relabel seeds based on pairwiseaffinity (step 1204). Then, the mechanism merges regions correspondingto the seeds with the same label (step 1205). It should be appreciatedthat due to the seed relabeling performed by the mechanisms of theillustrative embodiments, the split lesion mask output in step 1205 willnot have the over-splitting problems associated with the watershedtechnique due to the wrong labels being associated with the data pointsassociated with each of the lesion shapes, as previously discussedabove. Thereafter, the operation ends (step 1206).

Z-Wise Connection of Lesions

The above process for lesion splitting and seed relabeling may beperformed with regard to each of the two-dimensional images, or slices,of the input volume to thereby generate appropriately labeled lesionmasks for each of the lesions represented in the correspondingtwo-dimensional images. However, the input volume represents athree-dimensional representation of the biological entity's internalanatomical structure and lesions that may appear to be associated withthe same lesion, when considered in three-dimensions, may in fact beassociated with different lesions. Thus, in order to be able to properlyidentify separate lesions within the biological entity as represented inthree-dimensions of the input volume, the illustrative embodimentsprovide a mechanism for connecting two-dimensional lesions along thez-axis, i.e., in three-dimensions.

The mechanism that performs connection of the two-dimensional lesionsalong the z-axis, referred to as the z-wise connection of lesions,includes a logistic regression model that executes on the split lesionoutput generated by the mechanisms described above, to determinethree-dimensional z-wise lesion detection. The mechanism connects twolesions in neighboring image slices. Two lesions are connected when thelogistic regression model determines that they represent the samelesion. For example, for any two-dimensional lesions on neighboringimage slices, i.e. slices that have consecutively ordered z-axiscoordinates along a z-axis in a three-dimensionally organized collectionof the slices, the mechanism determines whether theses two-dimensionallesions belong to the same three-dimensional lesion or not, as will bedescribed hereafter.

FIGS. 13A-13C illustrate a process for z-wise connection of lesions inaccordance with one illustrative embodiment. FIG. 13A depicts a lesionmask input. FIG. 13B depicts the lesions after slice-wise lesionsplitting, which may employ the relabeling improved lesion splittingmechanisms of the illustrative embodiments as described previously. Asshown in FIGS. 13A-13B, slice 1310 has lesions 1311 and 1312, slice 1320has lesion 1321, and slice 1330 has lesions 1331 and 1332. The z-wiseconnection of lesions mechanism, i.e. the logistic regression model,executes on the split lesion masks for each neighboring pair of theslices in the input volume so as to compare each lesion in a given sliceto each lesion in the paired neighboring slice. For example, the z-wiseconnection of lesions mechanism compares lesion 1311 (lesion A) in slice1310 to lesion 1321 (lesion B) in slice 1320. For each comparison, themechanism treats each lesion as a set of voxels and determines theintersection between lesion A (set of voxels in lesion A) and lesion B(set of voxels in lesion B) with respect to the size of lesion A andwith respect to the size of lesion B. The z-wise connection of lesionsmechanism determines whether lesion A and lesion B are connected usingthe logistic regression model based on two overlap ratios, as follows:

${r_{0} = \frac{❘{A\bigcap B}❘}{\min\left( {{❘A❘},{❘B❘}} \right)}},{r_{1} = \frac{❘{A\bigcap B}❘}{\max\left( {{❘A❘},{❘B❘}} \right)}}$

Where |A| denotes the area of the circle represented by seed A, |B|denotes the area of the circle represented by seed B, and |A∩B| denotesthe area of the intersection of the circles represented by seed A andseed B. The mechanism uses these two ratios, as input features, to traina logistic regression model to determine the probability that lesion Aand lesion B are connected. That is, using a machine learning process,such as previously described above, the logistic regression model istrained on volumes of training images to generate predictions withregard to the probability that, in each pairwise combination of slicesin each training volume, whether a lesion in one slice is the same ordifferent lesion as a lesion represented in a neighboring slice. Thisprediction is compared to a ground truth indication of whether thelesions are the same or different lesions in order to generate a loss orerror. The operational parameters, e.g., coefficients or weights, of thelogistic regression model are then modified so as to reduce this loss orerror until a predetermined number of epochs of training have beenperformed or a predetermined stopping condition is satisfied.

Logistic regression models are widely used for solving binaryclassification problems. In the context of the illustrative embodiments,however, this logistic regression model predicts a probability for twocross-sections of lesions to be part of the same lesion. To this end,the logistic regression uses the two overlap ratios r₀ and r₁, asmentioned previously. Specifically, the logistic model learns tolinearly combine the two features as follows:

${f\left( {r_{0},{r_{1};c_{0}},c_{1},b} \right)} = \frac{1}{1 + {\exp\left( {{c_{0}r_{0}} + {c_{1}r_{1}} + b} \right)}}$

in which (c₀, c₁, b) are the operational parameters to be learned fromthe training volumes via the machine learning training operation.Notation-wise, r₀ and r₁ denote respectively minimum overlap ratio andmaximum overlap ratio. The state of the operational parameters aftertraining of the logistic regression model may be denoted by (c₀, c₁,b*). At inference time, i.e., after the training of the logisticregression model, when processing a new input volume of images (slices)a threshold t is set so that two cross-sections are deemed as belongingto the same lesion if, and only if, the relation f(r₀, r₁; c₀*, c₁*,b*)>t holds, i.e. the predicted probability is higher than the setthreshold.

There are two extreme cases. First, when the threshold t is set to 0,the z-wise connection mechanism of the illustrative embodiments alwaysdetermines that the lesions are the same lesion, i.e. that thecross-sections are connected. Then both the true positive rate and falsepositive rate will be 1. Second, when the threshold t is set to 1, thez-wise connection mechanism will not identify any cross-sections oflesions to be connected. In this case, both true positive rate and falsepositive rate will be 0. Therefore, only when a threshold t is in theinterval (0, 1), will the logistic regression model make a determinationas to whether lesion cross-sections are associated with the same lesionor not across neighboring slices. With an ideal logistic regressionmodel, the true positive rate is equal to 1 (all the true connectionsare identified) and at the same time, false positive rate is 0 (zerofalse connection is made).

Thus, once the logistic regression model is trained, new pairs of slicesmay be evaluated in this manner by calculating these ratios for thepairs and inputting them into the trained logistic regression model asinput features so as to generate predictions for each of these pairs andthen, if the predicted probability is equal to or greater than apredetermined threshold probability, the lesions A and B are consideredto be associated with the same lesion in three dimensions. Appropriaterelabeling of the lesions across the slices may then be performed so asto properly associate lesions in two-dimensional slices with the samelesion representation in other neighboring slices and thereby identifythree-dimensional lesions within the input volume.

There are rationales that support the two ratio input features used totrain the logistic regression model. For example, if lesions A and B aresufficiently different in size, then it is unlikely they are part of thesame lesions. Moreover, if lesions A and B do not intersect, as withlesion 1312 in slice 1310 and lesion 1321 in slice 1320, then thefeatures r₀, r₁ will have a zero value. As noted above, the logisticregression model performs the regression given the two feature valuesr₀, r₁ and outputs a probability value between 0 and 1 that represents alikelihood lesion A and lesion B are part of the same lesion.

FIG. 13C depicts a cross-section connection between slices in accordancewith one illustrative embodiment. As shown in FIG. 13C, the mechanismdetermines that lesion 1311 in slice 1310 and lesion 1321 in slice 1321are part of the same lesion by executing the trained logistic regressionmodel of the illustrative embodiments that predicts lesion commonalitybased on overlap ratios as discussed above. The mechanism alsodetermines that lesion 1321 in slice 1320 and lesion 1331 in slice 1330are part of the same lesion in a similar manner. Thus, the mechanismpropagates the intersecting lesions along the z-axis and performs z-axisconnection of lesions.

Based on the pairwise evaluation of slices in the input volume withregard to identifying z-wise connection of lesions across thetwo-dimensional slices, and the determination by the trained logisticregression model whether lesions are connected or not along the z-axis,relabeling of lesions may be performed so as to make sure that the samelabel for a lesion is applied to each of the lesion masks present ineach of the slices of the input volume, e.g., all of the lesion masksacross a set of slices in the input volume, where the lesions masks aredetermined by the logistic regression model to be associated with thesame lesion A, may be relabeled to specify that they are part of thesame lesion A. This may be performed for each lesion cross-section ineach of the slices of the input volume to thereby generatethree-dimensional associations of lesion masks for one or more lesionspresent in the input volume. This information may then be used torepresent or otherwise process lesions, such as in later downstreamcomputing system operations, in three-dimensions since all of thecross-sections associated with the same lesion are properly labeled inthe input volume.

FIGS. 14A and 14B illustrate results of the trained logistic regressionmodel in accordance with one illustrative embodiment. FIG. 14Aillustrates a receiver operating characteristic (ROC) curve for amaximum overlap ratio (r₀)+minimum overlap ratio (r₁) metric, and for amaximum overlap ratio metric. A ROC curve is a graphical plot thatillustrates the diagnostic ability of a binary classifier system as itsdiscrimination threshold is varied. The ROC curve is created by plottingthe true positive rate (TPR) against the false positive rate (FPR) atvarious threshold settings. FIG. 14B illustrates a precision-recallcurve for a maximum overlap ratio+minimum overlap ratio metric, and fora maximum overlap ratio metric. A precision-recall curve is a plot ofthe precision (y-axis) and the recall (x-axis) for different thresholds,much like the ROC curve, where precision is the fraction of relevantinstances among retrieved instances, and recall (or sensitivity) is thefraction of the total amount of relevant instances that were actuallyretrieved. As shown in these figures, the two-feature logistic modeloutperforms its one-feature counterpart. Both features thus bringvaluable information for this prediction task.

Looking at the maximum overlap ratio (r₀)+minimum overlap ratio (r₁)metric curve in FIG. 14A, it can be seen that with an appropriatethreshold t, the trained logistic regression model is able to produce atrue positive rate˜=95% at the cost of a false positive rate at roughly3%. Looking at FIG. 14B, the depicted plot assess the trained logisticregression model in terms of precision and recall, and shows that bothmeasures are able to achieve very good results with the selection of aproper threshold t.

FIG. 15 is a flowchart outlining an example operation of a mechanism forconnecting two-dimensional lesions along the z-axis in accordance withone illustrative embodiment. As shown in FIG. 15, the operation begins(step 1500), and the mechanism selects a first image X from the inputvolume (step 1501) and selects a first lesion A in image X (step 1502).In some illustrative embodiments, the images or slices in the inputvolume may be processed using the splitting and relabeling mechanismdescribed previously, however this is not required. To the contrary, themechanisms of the illustrative embodiments directed to the z-wiseconnection of lesions may in fact be performed with any input volume inwhich lesion masks have been identified.

The z-wise connection mechanism of the illustrative embodiments thenselect a first lesion B in an adjacent image Y (step 1503). Then, themechanism determines an intersect between lesion A and lesion B withrespect to lesion A and determines an intersect between lesion A andlesion B with respect to lesion B (step 1504). The mechanism determineswhether lesion A and lesion B belong to the same lesion based on the twointersect values by applying the trained logistic regression model tothe r₀ and r₁ features for the intersections of lesion A and lesion B togenerate a prediction or probability that lesion A and lesion B are thesame lesion and then compare the probability to a threshold probability(step 1505). Based on the results of this determination, thecross-sections of lesions in the images may be labeled or relabeled toindicate whether or not they are part of the same lesion.

The mechanism determines whether lesion B in image Y is the last lesionin image Y (step 1506). If lesion B is not the last lesion, then themechanism considers the next lesion B in the adjacent image Y (step1507), and operation returns to step 1504 to determine the intersectbetween the lesion A and the new lesion B.

If lesion B is the last lesion in the adjacent slice or image Y at step1506, then the mechanism determines whether lesion A is the last lesionin image X (step 1508). If lesion A is not the last lesion in image X,then the mechanism considers the next lesion A in image X (step 1509),and operation returns to step 1502 to consider the first lesion B in anadjacent image Y.

If lesion A is the last lesion in image X at step 1508, then themechanism determines whether image X is the last image to consider (step1510). If image X is not the last image, then the mechanism considersthe next image X (step 1511), and operation returns to step 1502 toconsider the first lesion A in the new image X.

If image X is the last image to consider at step 1510, then themechanism propagates intersecting lesions between images along thez-axis, where propagation means that the labels associated with the samelesion as determined through the above process are set to a same valueto indicate that they are part of the same lesion (step 1512). This isperformed for each separate lesion identified in the input volume suchthat cross-sections in each of the images that are associated with thesame lesion are appropriately labeled and thus, a three-dimensionalrepresentation of each lesion is generated through the z-wise connectionof the cross-sections. Thereafter, the operation ends (step 1513).

Contour Refinement

The above processes yield accurate results in terms of the number andrelative positions of lesions and connecting lesions acrosstwo-dimensional spaces (within an image or slice) and three-dimensionalspaces (across images or slices in an input volume). However, lesioncontours (boundaries) are not always well-defined and requireimprovement. The illustrative embodiments provide a mechanism forimproving lesion contour accuracy. This additional mechanism may beemployed with the above described mechanisms as part of lesionsegmentation, or may be employed in other illustrative embodiments whichdo not require the specific lesion detection, lesion splitting andrelabeling, and/or z-wise connection mechanisms described above.

Existing contour algorithms only work well when there is a lesion in themiddle of the anatomical structure with no surrounding lesions, butperform poorly when there are different circumstances leading to a“leaking” problem where two or more close-by lesions have theirinitially distinct contours merged into one single all-encompassingcontour, thereby completely erasing the benefits brought by the earliertwo dimensional lesion mask splitting. In some cases, where the lesionis at the vicinity of the anatomical structure boundary, e.g., the liverboundary, the contouring algorithm distinguishes between pixels of theanatomical structure relative to pixels for other anatomical structures,e.g., organs, in the image, rather than distinguishing one lesion fromanother, because these are most distinguishable by the contouringalgorithm.

The mechanisms of the illustrative embodiment in-paint areas that arenot of interest in the images or slices. FIG. 16 illustrates an examplewith contours for two lesions in the same image in accordance with anillustrative embodiment. On the left side of FIG. 16, an active contouralgorithm is used to determine contours 1611 and 1612 for the twolesions. Active contour algorithms are a class of algorithms that make acontour evolve iteratively to fit image content better.

In accordance with the illustrative embodiment, the mechanism in-paintswithin contour 1612 and non-liver tissue in the vicinity of contour1611, but not within contour 1611, where the in-painting means that thepixel values for pixels of the contour 1612 and within the contour 1612and healthy tissue (non-lesion tissue) in the vicinity of contour 1611are set to a specified value such that they all have a same value. Forexample, this value may be the average tissue value in the regionidentified as not being associated with a lesion, i.e. healthy tissue ofthe anatomical structure, e.g., the liver.

This in-painting may be performed with respect to a selected lesioncontour 1611 such that the in-painting is applied to the healthy tissueand other lesions, e.g., lesion 1612, in the image. In this way, thecontour and pixels associated with the selected lesion, e.g., 1611, areseparately considered from the other portions of the image whenre-evaluating the contour 1611. The contour 1611 may then bere-evaluated and a determination may be made as to whether there-evaluation of the contour 1611 results in an improved definition ofthe contour 1611. That is, an initial determination of the contrast andvariance between pixels associated with the selected lesion contour 1611and pixels near the selected lesion contour 1611 may be generated. Aftercalculating this contrast and variance prior to in-painting, thein-painting may be performed with respect to the selected lesion 1611such that pixels associated with other lesion contours, e.g., 1612, andareas of the anatomical structure representing healthy tissues in theimage, are in-painted with an average pixel intensity value of thehealthy tissue.

The variance of a set of values is determined as follows. Consider avoxel set, comprising of, say n voxels. First, the arithmetic average iscomputed by summing up their intensity values and then dividing theresulting sum by n. This is denoted the resulting quantity by A. Second,these voxel values are squared individually and then the arithmeticaverage is computed. The result is denoted as B. The variance is thendefined as B−A*A, i.e., the difference between B and the squared A.

Thus, the variance of a set of n values {x₁, . . . , x_(n)} is definedas follows:

$\frac{\Sigma_{i = 1}^{n}x_{i}^{2}}{n} - {\left( \frac{\Sigma_{i = 1}^{n}x_{i}}{n} \right)^{2}.}$

The variance is computed between voxels inside and outside a givencontour. Voxels inside a counter are those enclosed by the contour, andvoxels outside refer to those outside the contour but remaining within apre-determined distance away from the contour.

The mechanism recalculates the contour 1611 of the selected lesion afterthe in-painting using an active contouring algorithm as previouslydescribed above, and recalculates the contrast and/or variance of thenew contour 1611 to determine if these values have improved (highercontrast value or lower variance value inside and/or outside of thelesion). If the contrast and variance have improved, then the newlycalculated contour 1611 is maintained as the contour of thecorresponding lesion. This process may then be performed with regard tothe lesion 1612 as the selected lesion by then in-painting the pixelsassociated with lesion 1611 and the health tissue near contour 1612. Inthis way, each lesion is separately evaluated to generate a contour forthe lesion and thereby prevent leakage of lesions into each other.

The mechanism for calculating the contours of the lesion afterin-painting may be based on the Chan-Vese Segmentation algorithm whichis designed to segment objects without clearly defined boundaries. Thisalgorithm is based on level sets that are evolved iteratively tominimize an energy, which is defined by weighted values corresponding tothe sum of differences intensity from the average value outside thesegmented region, the sum of differences from the average value insidethe segmented region, and a term which is dependent on the length of theboundary of the segmented region. Initialization is done usingpartitioned detection map (solve energy local minima issues).

Once the mechanism has a segmentation, the mechanism initializes thecontour with previous estimates and determines whether the new contouris better, e.g., the contrast and variance of the contour are improved.If the original contour is better, then the original is maintained. Ifthe new contour is better, e.g., the contrast and variance of thecontour are improved, then the mechanism uses the new contour. In someillustrative embodiments, the mechanism determines which contour isbetter based on a homogenous area and computing a variance. If thevariance is reduced both inside and outside of the contour, then themechanism uses the new contour; otherwise, the mechanism uses the oldcontour. In another illustrative embodiment, the mechanism determineswhether contrast (mean inside the contour versus mean in the vicinity ofthe contour) is improved. Other techniques using different measures maybe used to choose between the old contour and the new contour withoutdeparting from the spirit and scope of the illustrative embodiments.

FIG. 17 is a flowchart outlining an example operation of a mechanism forslice-wise contour refinement in accordance with an illustrativeembodiment. As shown in FIG. 17, the operation begins for a givencontour in an image segmented to show lesions, such as in a liver, forexample (step 1700), and the mechanism determines a first contrast andvariance for an initial contour (step 1701). The mechanism in-paints thelesion pixels (or voxels in three dimensions) in the vicinity of thelesion (step 1702). The mechanism then determines a contour around thelesion (step 1703). Then, the mechanism determines a second contrast andvariance for the new contour (step 1704). The mechanism determineswhether the second contrast and variance represent an improvementcompared to the first contrast and variance (step 1705). If the secondcontrast and variance represent an improvement, then the mechanism usesthe updated contour to represent the lesion (step 1706). Thereafter,operation ends (step 1708).

If the second contrast and variance do not represent an improvement instep 1705, then the mechanism reverts to the initial contour (step1707). Thereafter, operation ends (step 1708). This process may berepeated for each lesion identified in the input slices and/or an inputvolume so as to recalculate the contours and improve the contoursassociated with each of the lesions present in the image/input volume.

False Positive Removal

After performing lesion segmentation to generate a list of lesions andtheir contours, the AI pipeline 100 performs a false positive stage ofprocessing 150 to remove lesions from the list of lesions that arefalsely indicated. This false positive stage 150 may take many forms toreduce the number of falsely identified lesions in the list of lesions,e.g., the contours and map 135 in FIG. 1 output by the liver/lesiondetection logic 130 which are then merged by segmentation and relabelingperformed in lesion segmentation logic 140. The following descriptionwill set forth a novel false positive removal mechanism which may beused to perform this false positive removal, but this particular falsepositive removal is not required. Moreover, the false positive removalmechanisms described hereafter may be used separately from the othermechanisms described above and may be applied to any listing of objectsidentified in images, with the illustrative embodiments utilizing suchfalse positive removal specifically with lesions in medical images. Thatis, the false positive removal mechanisms described in this section maybe implemented separately and distinctly from the other mechanismsdescribed herein above.

For purposes of illustration, it will be assumed that the false positiveremoval mechanisms are implemented as part of the AI pipeline 100 and aspart of the false positive removal logic 150 of the AI pipeline 100.Thus, in the false positive stage 150, the false positive removalmechanisms described in this section operate on the listing of lesionsthat results from the liver/lesion detection logic and the segmentationand relabeling of lesions, taking into account the three-dimensionalnature of the input volume with the z-wise connection of lesions andcontour refinement described above. This listing 148 in FIG. 1 is inputto the false positive removal logic stage 150 which processes thelisting 148 in the manner described hereafter and outputs a filtered ormodified list of lesions, in which falsely identified lesions areminimized in the modified list of lesions, to a lesion classificationstage 160. The lesion classification stage then classifies the variouslesions indicated in the modified list of lesions.

That is, the capturing of all lesions in the previous stages of the AIpipeline 100 may lead to increased sensitivity settings that cause theAI pipeline 100 to mis-identify pixels that do not actually representlesions as being part of a lesion. As a result, there may be falsepositives that should be removed. The false positive stage 150 compriseslogic that operates on the list of lesions and their contours to removefalse positives. It should be appreciated that such false positiveremoval must also balance the risk that, at the exam (set of inputvolumes level as opposed to the lesion level, removal of falsepositives, if not done appropriately, may result in lesions goingundetected. This may be problematic because the physician and patientmay not be made aware of lesions that need treatment. It should beappreciated that an exam can theoretically contain several volumes ofimages for the same patient. However, because in some illustrativeembodiments in which there is a single phase detection implemented AIpipeline, only one volume of images is processed, it is assumed that theprocessing is performed with regard to a single volume. For the sake ofclarity, “patient level” is used in place of “exam level” hereafter asthis is what is of interest to the illustrative embodiment (does thepatient have a lesion or not). It should be appreciated that in otherillustrative embodiments, the operations described herein may beextended to an exam level, in which multiple volumes of images for thesame patient may be evaluated.

With the illustrative embodiments, given the outputs of the prior stageof the AI pipeline 100 (slices, masks, lesions, lesion and anatomicalstructure contours, etc.) as inputs 148 to the false positive removalstage 150, the false positive removal stage 150 operates at a highlyspecific operating point at a patient level (the input volume level) inorder to admit only a few patient level false positives (normalpatients/volume where at least one lesion is detected). This point canbe retrieved from the analysis of a patient receiver operatingcharacteristic (ROC) (patient level sensitivity vs patient levelspecificity) analysis. For those volumes where using the highly specificoperating point, referred to herein as the patient level operating pointOP_(patient), yields at least some lesions, a more sensitive operatingpoint is used at a lesion level, referred to herein as the lesion leveloperating point OP_(lesion). The lesion level operating pointOP_(lesion) can be identified from the analysis of a lesion level ROCcurve (lesion sensitivity vs lesion specificity) in order to maximizethe number of lesions being kept.

The two operating points, i.e. OP_(patient) and OP_(lesion), may beimplemented in one or more trained ML/DL computer models. The one ormore trained ML/DL computer models are trained to classify the inputvolume and/or its listing of lesions (results of the segmentation logic)as to whether the identified lesions are true lesions or false lesions,i.e. true positives or false positives. The one or more trained ML/DLcomputer models may be implemented as binary classifiers, where theoutput indicates for each lesion whether it is a true positive or afalse positive. The set of outputs comprising the binary classificationfor all of the lesions in the input listing of lesions may be used tofilter the listing of lesions to remove the false positives. In oneillustrative embodiment, the one or more trained ML/DL computer modelsfirst implement the patient level operating point to determine if theresults of the classification indicate any of the lesions in the listingof lesions to be true positives while filtering out false positives. Ifthere are any true positives left in the first filtered listing oflesions after the patient level (input volume level) filtering, then thelesion level operating point is used to filter out the remaining falsepositives, if any. As a result, a filtered listing of lesions isgenerated which minimizes false positives.

The implementation of the operating points may be with regard to asingle trained ML/DL computer model or multiple trained ML/DL computermodels. For example, using a single trained ML/DL computer model, theoperating points may be settings of operating parameters of the ML/DLcomputer model which may be switched dynamically. For example, theinputs to the ML/DL computer model may be processed using the patientlevel operating point to generate a result indicating whether or not thelisting of lesions includes a true positive after classification of eachof the lesions and then, if it does, the operating point of the ML/DLcomputer model may be switched to the lesion level operating point andthe input processed again with the false positives of each pass throughthe ML/DL computer model being removed from the final listing of lesionsoutput by the false positive removal stage. Alternatively, in someillustrative embodiments, two separate ML/DL computer models may betrained, one for the patient level operating point and one for thelesion level operating point, such that the results of the first ML/DLcomputer model indicating at least one true positive causes processingof the input through the second ML/DL computer model and the falsepositives identified by both models being removed from the final listingof lesions output by the false positive removal stage of the AIpipeline.

The training of the ML/DL computer model(s) may involve a machinelearning training operation in which the ML/DL computer model processestraining inputs comprising volumes of images and corresponding listingsof lesions, where the listing of lesions includes lesion masks orcontours, to generate a classification for each lesion in the image asto whether it is a true or false positive. The training input further isassociated with ground truth information indicating whether the imageincludes a lesion or not which can then be used to evaluate the outputgenerated by the ML/DL computer model to determine a loss or error andthen modify the operating parameters of the ML/DL computer model toreduce the determined loss/error. In this way, the ML/DL computer modellearns features of the input that are representative of true/falsepositive lesion detection. This machine learning may be performed withregard to each of the operating points, i.e. OP_(patient) andOP_(lesion), such that the operating parameters of the ML/DL computermodel are learned taking into account the patient levelsensitivity/specificity and/or lesion level sensitivity/specificity.

In classifying lesions as to whether or not they are true positives orfalse positives, an input volume (representing the patient at the“patient level”) is considered to be positive if it contains at leastone lesion. An input volume is considered to be negative if it containsno lesions. With this in mind, a true positive is defined as a positiveinput volume, i.e. an input volume that has at least one findingclassified as a lesion that is actually a lesion. A true negative isdefined as a negative input volume, i.e. an input volume that has nolesion, and where no finding has been classified as a lesion. A falsepositive is defined as a negative input volume in which there is nolesion, however the input indicates a lesion in the findings, i.e. theAI pipeline lists a lesion when one is not present. A false negative isdefined as a positive input volume that has a lesion, but the AIpipeline does not indicate a lesion in the findings. The trained ML/DLcomputer model classifies lesions in the input as to whether they aretrue positives or false positives. False positives are filtered out ofthe output generated by the false positive removal. The detection offalse positives is performed at different levels ofsensitivity/specificity at a patient level and lesion level, i.e. thetwo different operating points.

The two different operating points for patient level and lesion levelmay be determined based on a ROC curve analysis. ROC curves may becomputed using ML/DL computer model validation data composed of severalinput volumes (e.g., several input volumes corresponding to differentpatient exams) that may contain some lesions (between 0 and K lesion(s)per exam). The input to the trained ML/DL computer model(s), or“classifier(s),” are previously detected findings in the input that areeither actual lesions or false positives, e.g., output of the lesiondetection and segmentation stages of the AI pipeline. The firstoperating point, i.e. the patient level operating point OP_(patient), isdefined to maintain at least X % of the lesions identified as truepositives, meaning that almost all the true positives are kept whileremoving some false positives. The value of X may be set based on theanalysis of the ROC curve and may be any suitable value for theparticular implementation. In one illustrative embodiment, the value ofX is set to 98% such that almost all of the true positives aremaintained while some false positives are removed.

The second operating point, i.e. the lesion level operating pointOP_(lesion), is defined such that lesion sensitivity is above the lesionsensitivity obtained for the first operating point, i.e. patient leveloperating point OP_(patient), and such that the specificity is above Y%, where Y depends on the actual performance of the trained ML/DLcomputer model. In one illustrative embodiment, Y is set to 30%. Anexample of ROC curves for patient level and lesion level operating pointdetermination is shown in FIG. 18A. As shown in FIG. 18A, the lesionlevel operating point is selected along the lesion level ROC curve suchthat the lesion sensitivity is above the lesion sensitivity for thepatient level operating point.

FIG. 18B is an example flow diagram of an operation for performing falsepositive removal based on patient and lesion level operating points inaccordance with one illustrative embodiment. As shown in FIG. 18B, theresults of the segmentation stage logic of the AI pipeline are input1810 to a first trained ML/DL computer model 1820 which implements thefirst operating point. The input 1810 includes the input volume (orvolume of images (VOI)) and the listing of lesions, which includes thelesion masks or contour data specifying pixels or voxels that correspondto each of the lesions identified in the image data of the volume ofimages and the labels associated with these pixels specifying whichlesions they correspond to three-dimensional space of the input volume,i.e. the output of the segmentation, z-wise connection, and contourrefinement described previously. The input may be denoted as set S. Thefirst trained ML/DL computer model 1820 implements the patient leveloperation point in its training so as to classify features extractedfrom the input with X %, e.g., 98%, of true positives being maintainedin the resulting filtered listing of lesions generated by theclassifications of the trained ML/DL computer model 1820 and some of thefalse positives being removed in the resulting listing. The resultinglisting comprises a subset S⁺ containing the true positive lesionsclassified by the first ML/DL computer model 1820 and a subset S⁻containing the false positive lesions classified by the first ML/DLcomputer model 1820.

The false positive removal logic further comprises true positiveevaluation logic 1830 which determines whether the subset of truepositives output by the first ML/DL computer model 1820 is empty or not.That is, the true positive evaluation logic 1830 determines if noelement from S is classified as a true lesion by the first ML/DLcomputer model 1820. If the subset of true positives is empty, the truepositive evaluation logic 1830 causes the true positive subset S⁺ to beoutput as the filtered listing of lesions 1835, i.e. no lesions will beidentified in the output sent to the lesion classification stage of theAI pipeline. If the true positive evaluation logic 1830 determines thatthe subset of true positives S⁺ is not empty, then a second ML/DLcomputer model 1840 is executed on the input S, where this second ML/DLcomputer model 1840 implements the second operating point in itstraining, i.e. the lesion level operating point OP_(lesion). It shouldbe appreciated that while two ML/DL computer models 1820 and 1840 areshown for ease of explanation, as noted above, these two operatingpoints may be implemented in different sets of trained operatingparameters for configuring the same ML/DL computer model such that thesecond ML/DL computer model may be a processing of the input S with thesame ML/DL computer model as 1820 but with different operationalparameters corresponding to the second operating point.

The second ML/DL computer model 1840 processes the input with thetrained operational parameters corresponding to the second operatingpoint to again generate classifications of lesions as to whether or notthey are true positives or false positives. The result is a subset S′⁺containing the predicted lesions (true positives) and a subset S′⁻containing the predicted false positives. The filtered listing oflesions 1845 is then output as the subset S′⁺, thereby effectivelyeliminating the false positives specified in the subset S′⁻.

The example embodiments shown in FIGS. 18A and 18B are described interms of patient level and lesion level operating points. It should beappreciated that the mechanisms of the false positive removal may beimplemented with various different levels of operating points. Forexample, a similar operation may be performed for image volume level andvoxel level operating points in a “voxel-wise” false positive removaloperation. FIG. 18C is an example flow diagram of an operation forperforming voxel-wise false positive removal based on input volume leveland voxel level operating points in accordance with one illustrativeembodiment. The operation in FIG. 18C is similar to that of FIG. 18B butwith the operations being performed with regard to voxels in the inputset S. With the voxel-wise false positive removal, the first operatingpoint may again be a patient level, or input volume level, operatingpoint, while the second operating point may be at voxel level operatingpoint OP_(voxel). In this case, true positives and false positives areevaluated at the voxel level such that if any voxel is indicated to beassociated with a lesion, and it is in fact associated with a lesion, itis a true positive, but if a voxel is indicated as being associated witha lesion and it is not in fact associated with a lesion, it is regardedas a false positive. Appropriate settings of the operating points may begenerated again based on corresponding ROC curves such that similarbalances between sensitivity and specificity are achieved as describedabove.

It should also be appreciated that while the above illustrativeembodiments of the false positive removal mechanisms assume a singleinput volume from a patient exam, the illustrative embodiments may beapplied to any grouping of one or more images (slices). For example, thefalse positive removal may be applied to a single slice, a set of slicessmaller than an input volume, or even multiple input volumes from thesame exam.

FIG. 19 is a flowchart outlining an example operation of false positiveremoval logic of an AI pipeline in accordance with one illustrativeembodiment. As shown in FIG. 19, the operation starts (step 1900) withreceiving the input S from the previous stage of the AI pipeline, wherethe input may include, for example, the input volume of images and thecorresponding listing of lesions including masks, contours, etc. (step1910). The input is processed by a first trained ML/DL computer modelthat is trained implementing a first operating point, e.g., patientlevel operating point that is relatively more highly specific and lesssensitive, to generate a first set of classifications for lesionscomprising a true positive subset and a false positive subset (step1920). A determination is made as to whether the true positive subset isempty (step 1930). If the true positive subset is empty, then theoperation outputs the true positive subset as the filtered listing oflesions (step 1940) and the operation terminates. If the true positivesubset is not empty, then the input S is processed by a second ML/DLcomputer model that is trained implementing a second operating pointthat is relatively more sensitive and less specific than the firstoperating point (step 1950). As noted above, in some illustrativeembodiments, the first and second ML/DL computer model may be the samemodel but configured with different operating parameters correspondingto the different training implementing the different operating points.The result of the processing of the second ML/DL computer model is asecond set of classifications for lesions comprising a second truepositive subset and a second false positive subset. The second truepositive subset is then output as the filtered listing of lesions (step1960) and the operation terminates.

Example Computer System Environment

The illustrative embodiments may be utilized in many different types ofdata processing environments. In order to provide a context for thedescription of the specific elements and functionality of theillustrative embodiments, FIGS. 20 and 21 are provided hereafter asexample environments in which aspects of the illustrative embodimentsmay be implemented. It should be appreciated that FIGS. 20 and 21 areonly examples and are not intended to assert or imply any limitationwith regard to the environments in which aspects or embodiments of thepresent invention may be implemented. Many modifications to the depictedenvironments may be made without departing from the spirit and scope ofthe present invention.

FIG. 20 depicts a schematic diagram of one illustrative embodiment of acognitive system 2000 implementing a request processing pipeline 2008,which in some embodiments may be a question answering (QA) pipeline,treatment recommendation pipeline, medical imaging augmentationpipeline, or any other artificial intelligence (AI) or cognitivecomputing based pipeline that processes a request using complexartificial intelligence mechanism that approximate human thoughprocesses with regard to a result generated, but through differentcomputer specific processes. For purposes of the present description, itwill be assumed that the request processing pipeline 2008 is implementedas a QA pipeline that operates on structured and/or unstructuredrequests in the form of input questions. One example of a questionprocessing operation which may be used in conjunction with theprinciples described herein is described in U.S. Patent ApplicationPublication No. 2011/0125734, which is herein incorporated by referencein its entirety.

The cognitive system 2000 is implemented on one or more computingdevices 2004A-D (comprising one or more processors and one or morememories, and potentially any other computing device elements generallyknown in the art including buses, storage devices, communicationinterfaces, and the like) connected to the computer network 2002. Forpurposes of illustration only, FIG. 20 depicts the cognitive system 2000being implemented on computing device 2004A only, but as noted above thecognitive system 2000 may be distributed across multiple computingdevices, such as a plurality of computing devices 2004A-D. The network2002 includes multiple computing devices 2004A-D, which may operate asserver computing devices, and 2010-2012 which may operate as clientcomputing devices, in communication with each other and with otherdevices or components via one or more wired and/or wireless datacommunication links, where each communication link comprises one or moreof wires, routers, switches, transmitters, receivers, or the like. Insome illustrative embodiments, the cognitive system 2000 and network2002 enables question processing and answer generation (QA)functionality for one or more cognitive system users via theirrespective computing devices 2010-2012. In other embodiments, thecognitive system 2000 and network 2002 may provide other types ofcognitive operations including, but not limited to, request processingand cognitive response generation which may take many different formsdepending upon the desired implementation, e.g., cognitive informationretrieval, training/instruction of users, cognitive evaluation of data,or the like. Other embodiments of the cognitive system 2000 may be usedwith components, systems, sub-systems, and/or devices other than thosethat are depicted herein.

The cognitive system 2000 is configured to implement a requestprocessing pipeline 2008 that receive inputs from various sources. Therequests may be posed in the form of a natural language question,natural language request for information, natural language request forthe performance of a cognitive operation, or the like. For example, thecognitive system 2000 receives input from the network 2002, a corpus orcorpora of electronic documents 2006, cognitive system users, and/orother data and other possible sources of input. In one embodiment, someor all of the inputs to the cognitive system 2000 are routed through thenetwork 2002. The various computing devices 2004A-D on the network 2002include access points for content creators and cognitive system users.Some of the computing devices 2004A-D include devices for a databasestoring the corpus or corpora of data 2006 (which is shown as a separateentity in FIG. 20 for illustrative purposes only). Portions of thecorpus or corpora of data 2006 may also be provided on one or more othernetwork attached storage devices, in one or more databases, or othercomputing devices not explicitly shown in FIG. 20. The network 2002includes local network connections and remote connections in variousembodiments, such that the cognitive system 2000 may operate inenvironments of any size, including local and global, e.g., theInternet.

In one embodiment, the content creator creates content in a document ofthe corpus or corpora of data 2006 for use as part of a corpus of datawith the cognitive system 2000. The document includes any file, text,article, or source of data for use in the cognitive system 2000.Cognitive system users access the cognitive system 2000 via a networkconnection or an Internet connection to the network 2002, and inputquestions/requests to the cognitive system 2000 that areanswered/processed based on the content in the corpus or corpora of data2006. In one embodiment, the questions/requests are formed using naturallanguage. The cognitive system 2000 parses and interprets thequestion/request via a pipeline 2008, and provides a response to thecognitive system user, e.g., cognitive system user 2010, containing oneor more answers to the question posed, response to the request, resultsof processing the request, or the like. In some embodiments, thecognitive system 2000 provides a response to users in a ranked list ofcandidate answers/responses while in other illustrative embodiments, thecognitive system 2000 provides a single final answer/response or acombination of a final answer/response and ranked listing of othercandidate answers/responses.

The cognitive system 2000 implements the pipeline 2008 which comprises aplurality of stages for processing an input question/request based oninformation obtained from the corpus or corpora of data 2006. Thepipeline 2008 generates answers/responses for the input question orrequest based on the processing of the input question/request and thecorpus or corpora of data 2006.

In some illustrative embodiments, the cognitive system 2000 may be theIBM Watson™ cognitive system available from International BusinessMachines Corporation of Armonk, N.Y., which is augmented with themechanisms of the illustrative embodiments described hereafter. Asoutlined previously, a pipeline of the IBM Watson™ cognitive systemreceives an input question or request which it then parses to extractthe major features of the question/request, which in turn are then usedto formulate queries that are applied to the corpus or corpora of data2006. Based on the application of the queries to the corpus or corporaof data 2006, a set of hypotheses, or candidate answers/responses to theinput question/request, are generated by looking across the corpus orcorpora of data 2006 for portions of the corpus or corpora of data 2006(hereafter referred to simply as the corpus 2006) that have somepotential for containing a valuable response to the inputquestion/response (hereafter assumed to be an input question). Thepipeline 2008 of the IBM Watson™ cognitive system then performs deepanalysis on the language of the input question and the language used ineach of the portions of the corpus 2006 found during the application ofthe queries using a variety of reasoning algorithms.

The scores obtained from the various reasoning algorithms are thenweighted against a statistical model that summarizes a level ofconfidence that the pipeline 2008 of the IBM Watson™ cognitive system2000, in this example, has regarding the evidence that the potentialcandidate answer is inferred by the question. This process is berepeated for each of the candidate answers to generate ranked listing ofcandidate answers which may then be presented to the user that submittedthe input question, e.g., a user of client computing device 2010, orfrom which a final answer is selected and presented to the user. Moreinformation about the pipeline 2008 of the IBM Watson™ cognitive system2000 may be obtained, for example, from the IBM Corporation website, IBMRedbooks, and the like. For example, information about the pipeline ofthe IBM Watson™ cognitive system can be found in Yuan et al., “Watsonand Healthcare,” IBM developerWorks, 2011 and “The Era of CognitiveSystems: An Inside Look at IBM Watson and How it Works” by Rob High, IBMRedbooks, 2012.

As noted above, while the input to the cognitive system 2000 from aclient device may be posed in the form of a natural language question,the illustrative embodiments are not limited to such. Rather, the inputquestion may in fact be formatted or structured as any suitable type ofrequest which may be parsed and analyzed using structured and/orunstructured input analysis, including but not limited to the naturallanguage parsing and analysis mechanisms of a cognitive system such asIBM Watson™, to determine the basis upon which to perform cognitiveanalysis and providing a result of the cognitive analysis. For example,a physician, patient, or the like may issue a request via their clientcomputing device 2010 to the cognitive system 2000 for a particularmedical imaging based operation, e.g., “identify liver lesions presentin patient ABC” or “provide treatment recommendations for patient” or“identify changes in liver lesions for patient ABC”, or the like. Inaccordance with the illustrative embodiments, such requests may bespecifically directed to cognitive computer operations that employ thelesion detection and classification mechanisms of the illustrativeembodiments to provide a listing of lesions, contours of lesions,classification of lesions, and contours of the anatomical structure ofinterest, upon which the cognitive system 2000 operates to provide acognitive computing output. For example, the request processing pipeline2008 may process a request such as “identify liver lesions present inpatient ABC” to parse this request and thereby identify the anatomicalstructure of interest to be the “liver”, the particular input volumebeing a medical imaging volume for patient “ABC”, and that “lesions” inthe anatomical structure are to be identified. Based on this parsing,the particular medical imaging volume corresponding to patient “ABC” maybe retrieved from the corpus 2006 and input to the lesion detection andclassification AI pipeline 2020 which operates on this input volume aspreviously described above so as to identify the listing of liverlesions which is output to the cognitive computing system 2000 forfurther evaluation through the request processing pipeline 2008, forgenerating a medical imaging viewer application output, or the like.

As shown in FIG. 20, one or more of the computing devices, e.g., server2004, may be specifically configured to implement the lesion detectionand classification AI pipeline 2020, such as the AI pipeline 100 in FIG.1, for example. The configuring of the computing device may comprise theproviding of application specific hardware, firmware, or the like tofacilitate the performance of the operations and generation of theoutputs described herein with regard to the illustrative embodiments.The configuring of the computing device may also, or alternatively,comprise the providing of software applications stored in one or morestorage devices and loaded into memory of a computing device, such asserver 2004, for causing one or more hardware processors of thecomputing device to execute the software applications that configure theprocessors to perform the operations and generate the outputs describedherein with regard to the illustrative embodiments. Moreover, anycombination of application specific hardware, firmware, softwareapplications executed on hardware, or the like, may be used withoutdeparting from the spirit and scope of the illustrative embodiments.

It should be appreciated that once the computing device is configured inone of these ways, the computing device becomes a specialized computingdevice specifically configured to implement the mechanisms of theillustrative embodiments and is not a general purpose computing device.Moreover, as described herein, the implementation of the mechanisms ofthe illustrative embodiments improves the functionality of the computingdevice and provides a useful and concrete result that facilitatesautomatic lesion detection in an anatomical structure of interest, aswell as classification of such lesions, which reduces error and improvesefficiency relative to manual processes.

As noted above, the mechanisms of the illustrative embodiments utilizespecifically configured computing devices, or data processing systems,to perform the operations for performing anatomical structureidentification, lesion detection and classification. These computingdevices, or data processing systems, may comprise various hardwareelements which are specifically configured, either through hardwareconfiguration, software configuration, or a combination of hardware andsoftware configuration, to implement one or more of thesystems/subsystems described herein. FIG. 21 is a block diagram of justone example data processing system in which aspects of the illustrativeembodiments may be implemented. Data processing system 2100 is anexample of a computer, such as server 2004 in FIG. 20, in which computerusable code or instructions implementing the processes and aspects ofthe illustrative embodiments of the present invention may be locatedand/or executed so as to achieve the operation, output, and externaleffects of the illustrative embodiments as described herein.

In the depicted example, data processing system 2100 employs a hubarchitecture including north bridge and memory controller hub (NB/MCH)2102 and south bridge and input/output (I/O) controller hub (SB/ICH)2104. Processing unit 2106, main memory 2108, and graphics processor2110 are connected to NB/MCH 2102. Graphics processor 2110 may beconnected to NB/MCH 2102 through an accelerated graphics port (AGP).

In the depicted example, local area network (LAN) adapter 2112 connectsto SB/ICH 2104. Audio adapter 2116, keyboard and mouse adapter 2120,modem 2122, read only memory (ROM) 2124, hard disk drive (HDD) 2126,CD-ROM drive 2130, universal serial bus (USB) ports and othercommunication ports 2132, and PCI/PCIe devices 2134 connect to SB/ICH2104 through bus 2138 and bus 2140. PCI/PCIe devices may include, forexample, Ethernet adapters, add-in cards, and PC cards for notebookcomputers. PCI uses a card bus controller, while PCIe does not. ROM 2124may be, for example, a flash basic input/output system (BIOS).

HDD 2126 and CD-ROM drive 2130 connect to SB/ICH 2104 through bus 2140.HDD 2126 and CD-ROM drive 2130 may use, for example, an integrated driveelectronics (IDE) or serial advanced technology attachment (SATA)interface. Super I/O (SIO) device 2136 may be connected to SB/ICH 2104.

An operating system runs on processing unit 2106. The operating systemcoordinates and provides control of various components within the dataprocessing system 2100 in FIG. 21. As a client, the operating system maybe a commercially available operating system such as Microsoft® Windows10®. An object-oriented programming system, such as the Java™programming system, may run in conjunction with the operating system andprovides calls to the operating system from Java™ programs orapplications executing on data processing system 200.

As a server, data processing system 2100 may be, for example, an IBMeServer™ System P® computer system, Power™ processor based computersystem, or the like, running the Advanced Interactive Executive (AIX⁺)operating system or the LINUX⁺ operating system. Data processing system2100 may be a symmetric multiprocessor (SMP) system including aplurality of processors in processing unit 2106. Alternatively, a singleprocessor system may be employed.

Instructions for the operating system, the object-oriented programmingsystem, and applications or programs are located on storage devices,such as HDD 2126, and may be loaded into main memory 2108 for executionby processing unit 2106. The processes for illustrative embodiments ofthe present invention may be performed by processing unit 2106 usingcomputer usable program code, which may be located in a memory such as,for example, main memory 2108, ROM 2124, or in one or more peripheraldevices 2126 and 2130, for example.

A bus system, such as bus 2138 or bus 2140 as shown in FIG. 21, may becomprised of one or more buses. Of course, the bus system may beimplemented using any type of communication fabric or architecture thatprovides for a transfer of data between different components or devicesattached to the fabric or architecture. A communication unit, such asmodem 2122 or network adapter 2112 of FIG. 21, may include one or moredevices used to transmit and receive data. A memory may be, for example,main memory 2108, ROM 2124, or a cache such as found in NB/MCH 2102 inFIG. 21.

As mentioned above, in some illustrative embodiments the mechanisms ofthe illustrative embodiments may be implemented as application specifichardware, firmware, or the like, application software stored in astorage device, such as HDD 2126 and loaded into memory, such as mainmemory 2108, for executed by one or more hardware processors, such asprocessing unit 2106, or the like. As such, the computing device shownin FIG. 21 becomes specifically configured to implement the mechanismsof the illustrative embodiments and specifically configured to performthe operations and generate the outputs described herein with regard tothe lesion detection and classification artificial intelligencepipeline.

Those of ordinary skill in the art will appreciate that the hardware inFIGS. 20 and 21 may vary depending on the implementation. Other internalhardware or peripheral devices, such as flash memory, equivalentnon-volatile memory, or optical disk drives and the like, may be used inaddition to or in place of the hardware depicted in FIGS. 20 and 21.Also, the processes of the illustrative embodiments may be applied to amultiprocessor data processing system, other than the SMP systemmentioned previously, without departing from the spirit and scope of thepresent invention.

Moreover, the data processing system 2100 may take the form of any of anumber of different data processing systems including client computingdevices, server computing devices, a tablet computer, laptop computer,telephone or other communication device, a personal digital assistant(PDA), or the like. In some illustrative examples, data processingsystem 2100 may be a portable computing device that is configured withflash memory to provide non-volatile memory for storing operating systemfiles and/or user-generated data, for example. Essentially, dataprocessing system 2100 may be any known or later developed dataprocessing system without architectural limitation.

As noted above, it should be appreciated that the illustrativeembodiments may take the form of an entirely hardware embodiment, anentirely software embodiment or an embodiment containing both hardwareand software elements. In one example embodiment, the mechanisms of theillustrative embodiments are implemented in software or program code,which includes but is not limited to firmware, resident software,microcode, etc.

A data processing system suitable for storing and/or executing programcode will include at least one processor coupled directly or indirectlyto memory elements through a communication bus, such as a system bus,for example. The memory elements can include local memory employedduring actual execution of the program code, bulk storage, and cachememories which provide temporary storage of at least some program codein order to reduce the number of times code must be retrieved from bulkstorage during execution. The memory may be of various types including,but not limited to, ROM, PROM, EPROM, EEPROM, DRAM, SRAM, Flash memory,solid state memory, and the like.

Input/output or I/O devices (including but not limited to keyboards,displays, pointing devices, etc.) can be coupled to the system eitherdirectly or through intervening wired or wireless I/O interfaces and/orcontrollers, or the like. I/O devices may take many different formsother than conventional keyboards, displays, pointing devices, and thelike, such as for example communication devices coupled through wired orwireless connections including, but not limited to, smart phones, tabletcomputers, touch screen devices, voice recognition devices, and thelike. Any known or later developed I/O device is intended to be withinthe scope of the illustrative embodiments.

Network adapters may also be coupled to the system to enable the dataprocessing system to become coupled to other data processing systems orremote printers or storage devices through intervening private or publicnetworks. Modems, cable modems and Ethernet cards are just a few of thecurrently available types of network adapters for wired communications.Wireless communication based network adapters may also be utilizedincluding, but not limited to, 802.11 a/b/g/n wireless communicationadapters, Bluetooth wireless adapters, and the like. Any known or laterdeveloped network adapters are intended to be within the spirit andscope of the present invention.

The description of the present invention has been presented for purposesof illustration and description, and is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the describedembodiments. The embodiment was chosen and described in order to bestexplain the principles of the invention, the practical application, andto enable others of ordinary skill in the art to understand theinvention for various embodiments with various modifications as aresuited to the particular use contemplated. The terminology used hereinwas chosen to best explain the principles of the embodiments, thepractical application or technical improvement over technologies foundin the marketplace, or to enable others of ordinary skill in the art tounderstand the embodiments disclosed herein.

1. A method, in a data processing system comprising at least oneprocessor and at least one memory, the at least one memory comprisinginstructions executed by the at least one processor to implement alesion detection and classification artificial intelligence (AI)pipeline, wherein the AI pipeline executes the method comprising:processing, by one or more first machine learning computer models of theAI pipeline, an input volume of medical images to determine whether ornot the input volume depicts a predetermined amount of an anatomicalstructure of interest; determining, by logic of the AI pipeline, whetherone or more predetermined criteria are satisfied by output of the one ormore first machine learning computer models, wherein the one or morepredetermined criteria comprises a predetermined amount of an anatomicalstructure of interest being depicted in the input volume; executing, bythe AI pipeline, in response to determining that the one or morepredetermined criteria are satisfied by the output of the one or morefirst machine learning computer models, a lesion detection operationthat detects whether the input volume of medical images includes thepresence of one or more lesions; and outputting, by the AI pipeline, anoutput based on results of the lesion detection for downstream computingsystem processing.
 2. The method of claim 1, wherein the one or morefirst machine learning computer models of the AI pipeline comprises abody part detection machine learning model that executes on the inputvolume, detects a portion of a biological entity's body represented inthe input volume, and determines whether the portion of the biologicalentity's body corresponds to a body part in which the anatomicalstructure of interest is present.
 3. The method of claim 2, wherein thebody part is a human abdomen, and the anatomical structure of interestis a human liver.
 4. The method of claim 1, wherein the one or morefirst machine learning computer models of the AI pipeline comprises aphase classification machine learning model and an anatomical structuredetection machine learning model that operate on the input volume,wherein the phase classification machine learning model determines aphase of medical imaging represented in each of the medical images ofthe input volume, and wherein the anatomical structure detection machinelearning model determines an amount of the anatomical structure ofinterest that is represented in the input volume.
 5. The method of claim4, wherein the one or more criteria further comprise the input volumehaving a single phase of medical imaging present in the input volume,and wherein based on results of an operation of the phase classificationmachine learning model on the input volume, and determining that aminimum threshold amount of the anatomical structure of interest ispresent in the input volume based on results of an operation of theanatomical structure detection machine learning model on the inputvolume.
 6. The method of claim 4, wherein, in response to results of theoperation of the phase classification machine learning model indicatingthat more than one phase of medical imaging is present in the inputvolume, selecting a subset of medical images in the input volume thatcorrespond to a phase of interest and executing the lesion detectionoperation comprises executing the lesion detection operation on only theselected subset of medical images in the input volume.
 7. The method ofclaim 6, wherein the phase of interest is one of a pre-contrast phase,an arterial contrast phase, portal/venous contrast phase, or a delayedphase.
 8. The method of claim 1, wherein determining whether one or morepredetermined criteria are satisfied by output of the one or more firstmachine learning computer models further comprises: executing, by axialscoring logic of the AI pipeline, an axial scoring operation on medicalimages in the input volume, wherein the axial scoring operationcomprises scoring images in the input volume in accordance with apredefined scoring algorithm to infer, from scores associated with theimages, axial scores for a most inferior slice in the volume (MISV) anda most superior slice in the volume (MSSV) and identifying a fraction ofthe anatomical structure of interest present in the input volume basedon the axial scores for the MISV and MSSV.
 9. The method of claim 1,wherein executing the lesion detection operation comprises processingthe input volume of medical images by one or more second machinelearning computer models of the AI pipeline to detect lesions present inone or more medical images of the input volume of medical images, andwherein the one or more second machine learning computer models of theAI pipeline comprises an ensemble of machine learning computer models,wherein each machine learning computer model in the ensemble of machinelearning computer models is trained by machine learning processes toprocess the input volume in a different manner from other machinelearning computer models of the ensemble of machine learning computermodels, to generate corresponding lesion detection predictions.
 10. Themethod of claim 9, wherein the ensemble of machine learning computermodels comprises a mask generation machine learning model, an inputvolume processing machine learning model, and a masked input volumeprocessing machine learning model, wherein the mask generation machinelearning model is trained, by a machine learning process, to generate amask corresponding to the anatomical structure of interest and appliesthe mask to the input volume to generate a masked input volume, whereinthe input volume processing machine learning model is trained, by amachine learning process, to generate a first lesion prediction outputbased on the input volume, and the masked input volume processingmachine learning model is trained, by a machine learning process, togenerate a second lesion prediction output based on the masked inputvolume.
 11. The method of claim 9, wherein: a first machine learningmodel of the ensemble is trained with a first loss function thatpenalizes errors in false negative classifications of lesions, and asecond machine learning model of the ensemble is trained with a secondloss function, different from the first machine learning model, thatpenalizes errors in false positive classifications of lesions.
 12. Themethod of claim 11, wherein logic of the ensemble applies a third lossfunction that compares a first lesion detection output of the firstmachine learning model to a second lesion detection output of the secondmachine learning model and operates to make the first lesion detectionoutput consistent with the second lesion detection output.
 13. Themethod of claim 1, wherein executing the lesion detection operationcomprises: processing the input volume of medical images by one or moresegmentation machine learning computer models of the AI pipeline toperform lesion segmentation on the one or more lesions that segmentsmedical images of the input volume into one or more contours to generatelesion segments; and processing the lesion segments by a z-wiseconnection machine learning model that combines subsets of the lesionsegments determined to be associated with a same lesion in threedimensional space.
 14. The method of claim 1, further comprising one ormore false positive removal machine learning computer models thatoperate on the one or more lesions to remove false positive detectionsof lesions from the one or more lesions.
 15. The method of claim 14,wherein the one or more false positive removal machine learning computermodels comprise one or more false positive removal machine learningcomputer models trained, by a machine learning process, based on twodifferent operating points, wherein a first operating point of the twodifferent operating points corresponds to a patient level operatingpoint and a second operating point of the two different operating pointscorresponds to a lesion level operating point.
 16. The method of claim1, wherein executing the lesion detection operation comprises:processing the one or more lesions by one or more fourth learningcomputer models that classify each of the lesions in the listing oflesions into one of a plurality of predetermined classes comprising abenign class, a malignant class, and an indeterminate class, to therebygenerate a classification for each lesion.
 17. The method of claim 16,further comprising: processing, by a downstream computing system, theoutput of the one or more lesions and corresponding classificationsassociated with each of the lesions to generate a visual representationof the one or more lesions in a graphical user interface.
 18. The methodof claim 1, wherein the input volume comprises medical imagescorresponding to at least one of a magnetic resonance imaging modalityor a computed tomography modality.
 19. A computer program productcomprising a computer readable storage medium having a computer readableprogram stored therein, wherein the computer readable program, whenexecuted on a computing device, causes the computing device to implementa lesion detection and classification artificial intelligence (AI)pipeline, wherein the AI pipeline executes to: process, by one or morefirst machine learning computer models of the AI pipeline, an inputvolume of medical images to determine whether or not the input volumedepicts a predetermined amount of an anatomical structure of interest;determine, by logic of the AI pipeline, whether one or morepredetermined criteria are satisfied by output of the one or more firstmachine learning computer models, wherein the one or more predeterminedcriteria comprises a predetermined amount of an anatomical structure ofinterest being depicted in the input volume; execute, by the AIpipeline, in response to determining that the one or more predeterminedcriteria are satisfied by the output of the one or more first machinelearning computer models, a lesion detection operation that detectswhether the input volume of medical images includes the presence of oneor more lesions; and output, by the AI pipeline, an output based onresults of the lesion detection for downstream computing systemprocessing.
 20. An apparatus comprising: a processor; and a memorycoupled to the processor, wherein the memory comprises instructionswhich, when executed by the processor, cause the processor to implementa lesion detection and classification artificial intelligence (AI),wherein the AI pipeline executes to: process, by one or more firstmachine learning computer models of the AI pipeline, an input volume ofmedical images to determine whether or not the input volume depicts apredetermined amount of an anatomical structure of interest; determine,by logic of the AI pipeline, whether one or more predetermined criteriaare satisfied by output of the one or more first machine learningcomputer models, wherein the one or more predetermined criteriacomprises a predetermined amount of an anatomical structure of interestbeing depicted in the input volume; execute, by the AI pipeline, inresponse to determining that the one or more predetermined criteria aresatisfied by the output of the one or more first machine learningcomputer models, a lesion detection operation that detects whether theinput volume of medical images includes the presence of one or morelesions; and output, by the AI pipeline, an output based on results ofthe lesion detection for downstream computing system processing.